User Guide

• 1: Setting Up Network PXE Boot
• 2: Kickstart Support in Photon OS
• 3: Kubernetes on Photon OS
• 3.1: Prerequisites
• 3.2: Running Kubernetes on Photon OS
• 3.2.1: System Information
• 3.2.2: Prepare the Hosts
• 3.2.3: Configure Kubernetes Services on the Master
• 3.2.4: Configure the Kubernetes services on Node
• 4: Photon NFS Utilities for Mounting Remote File Systems

1 - Setting Up Network PXE Boot

Photon OS supports the Preboot Execution Environment, or PXE, over a network connection. This document describes how to set up a PXE boot server to install Photon OS.

Server Setup

To set up a PXE server, you will need to have the following:

• A DHCP server to allow hosts to get an IP address.
• A TFTP server, which is a file transfer protocol similar to FTP with no authentication.
• Optionally, an HTTP server. The HTTP server will serve the RPMs yum repo, or you can use the official VMware Photon Packages repo. Also, this HTTP server can be used if you want to provide a kickstart config for unattended installation.

The instructions to set up the servers assume you have an Ubuntu 14.04 machine with a static IP address of 172.16.78.134.

DHCP Setup

• Install the DHCP server:

  sudo apt-get install isc-dhcp-server  
  

• Edit the Ethernet interface in /etc/default/isc-dhcp-server to INTERFACES="eth0"

• Edit the DHCP configuration in /etc/dhcp/dhcpd.conf to allow machines to boot and get an IP address via DHCP in the range 172.16.78.230 - 172.16.78.250, for example:

  subnet 172.16.78.0 netmask 255.255.255.0 {
    range 172.16.78.230 172.16.78.250;
    option subnet-mask 255.255.255.0;
    option routers 172.16.78.134;
    option broadcast-address 172.16.78.255;
    filename "pxelinux.0";
    next-server 172.16.78.134;
  }
  

• Restart the DHCP server:

  sudo service isc-dhcp-server restart
  

TFTP Setup

• Install the TFTP server:

  sudo apt-get install tftpd-hpa
  

• Enable the boot service and restart the service:

  sudo update-inetd --enable BOOT
  sudo service tftpd-hpa restart
  

Optional: HTTP server setup

This step is only needed if you are planning to serve the ks (kickstart) config file through this server; refer to Kickstart support for details.

• Serving your local yum repo.

You can install apache http web server:

sudo apt-get install apache2

Mount the Photon iso to get the RPMS repo and sample ks config file:

mkdir /mnt/photon-iso
sudo mount <photon_iso> /mnt/photon-iso/

Copy the RPMS repo:

cp -r /mnt/photon-iso/RPMS /var/www/html/

To support ks, you can copy the sample config file from the iso and edit it; refer to Kickstart support for details.

cp /mnt/photon-iso/isolinux/sample_ks.cfg /var/www/html/my_ks.cfg

PXE boot files setup

• Mount photon.iso to get Linux and initrd images:

mkdir /mnt/photon-iso
sudo mount <photon_iso> /mnt/photon-iso/

• Setting the PXE boot files:

wget https://www.kernel.org/pub/linux/utils/boot/syslinux/syslinux-6.03.tar.gz
tar -xvf syslinux-6.03.tar.gz
pushd /var/lib/tftpboot
cp -r /mnt/photon-iso/isolinux/* .
cp ~/syslinux-6.03/bios/com32/elflink/ldlinux/ldlinux.c32 .
cp ~/syslinux-6.03/bios/com32/lib/libcom32.c32 .
cp ~/syslinux-6.03/bios/com32/libutil/libutil.c32 .
cp ~/syslinux-6.03/bios/com32/menu/vesamenu.c32 .
cp ~/syslinux-6.03/bios/core/pxelinux.0 .
mkdir pxelinux.cfg
mv isolinux.cfg pxelinux.cfg/default

• Update repo param to point to http yum repo; you may pass official photon packages repo.

sed -i "s/append/append repo=http:\/\/172.16.78.134\/RPMS/g" menu.cfg
popd

• Optionally, you can add your ks config file; see Kickstart support for details.

2 - Kickstart Support in Photon OS

Photon OS works with kickstart for unattended, automated installations. The kickstart configuration file can be served through an HTTP server. You can also provide the kickstart configuration file through a secondary device or a CD-ROM attached to the host.

• Kickstart Capabilities
• Permitted JSON Fields
• Sample Configuration File
• Unattended Installation Through Kickstart

Kickstart Capabilities

Photon OS supports the following configurations with kickstart:

• Setting the hostname
• Setting the password
• Setting the disk to install
• Selecting whether to install the full or the minimal version of Photon OS
• Applying a post-installation script
• Adding public keys to allow the root account to log in through SSH

Permitted JSON Fields

The following is a list of allowed JSON fields and their descriptions:

            <p>{"mountpoint": "/root", "size": 128, "filesystem": "ext4","lvm":{"vg_name":"vg1", "lv_name":"root"}},

            <p>{"size": 128, "filesystem": "swap","lvm":{"vg_name":"vg2", "lv_name":"swap"}}

Sample Configuration File

Example kickstart configuration file:

{
    "hostname": "photon-machine",
    "password":
        {
            "crypted": false,
            "text": "changeme"
        },
    "disk": "/dev/sda",
    "partitions":[
		{
			"mountpoint":"/",
			"size":0,
			"filesystem":"ext4"
		},
		{
			"mountpoint":"/boot",
			"size":128,
			"filesystem":"ext4"
		},
		{
			"mountpoint":"/root",
			"size":128,
			"filesystem":"ext4"
		},
		{
			"size":128,
			"filesystem":"swap"
		}
		],
	"bootmode": "bios",
    "packagelist_file": "packages_minimal.json",
    "additional_packages": [
		"vim"
		],
    "postinstall": [
		"#!/bin/sh",
        "echo \"Hello World\" > /etc/postinstall"
        ],
    "public_key": "<ssh-key-here>",
    "linux_flavor": "linux",
    "network": {
        "type": "dhcp"
    }    
}

Installing Root Partition as LVM

In the kickstart file modify the partitions field to mount root partition as LVM.

For example:

"disk":"/dev/sda"
"partitions":[
   {
      "mountpoint":"/",
      "size":0,
      "filesystem":"ext4",
      "lvm":{
         "vg_name":"vg1",
         "lv_name":"rootfs"
      }
   },
   {
      "mountpoint":"/boot",
      "size":128,
      "filesystem":"ext4"
   },
   {
      "mountpoint":"/root",
      "size":128,
      "filesystem":"ext4",
      "lvm":{
         "vg_name":"vg1",
         "lv_name":"root"
      }
   },
   {
      "size":128,
      "filesystem":"swap",
      "lvm":{
         "vg_name":"vg2",
         "lv_name":"swap"
      }
   }
]

Note:

• vg_name : Volume Group Name
• lv_name : Logical Volume Name

In above example rootfs, root are logical volumes in the volume group vg1 and swap is logical volume in volume group vg2, physical volumes are part of disk /dev/sda.

Multiple disks are also supported. For example:

"disk": "/dev/sda"
"partitions":[
   {
      "mountpoint":"/",
      "size":0,
      "filesystem":"ext4",
      "lvm":{
         "vg_name":"vg1",
         "lv_name":"rootfs"
      }
   },
   {
      "mountpoint":"/boot",
      "size":128,
      "filesystem":"ext4"
   },
   {
      "disk":"/dev/sdb",
      "mountpoint":"/root",
      "size":128,
      "filesystem":"ext4",
      "lvm":{
         "vg_name":"vg1",
         "lv_name":"root"
      }
   },
   {
      "size":128,
      "filesystem":"swap",
      "lvm":{
         "vg_name":"vg1",
         "lv_name":"swap"
      }
   }
]

If disk name is not specified, the physical volumes will be part of the default disk: dev/sda.

In above example rootfs,root and swap are logical volumes in volume group vg1, physical volumes are in the disk /dev/sdb and partitions are present in /dev/sda.

Note: Mounting /boot partition as LVM is not supported.

Unattended Installation Through Kickstart

For an unattended installation, you pass the ks=<config_file> parameter to the kernel command. To pass the config file, there are three options:

1. Provide it in the ISO through a CD-ROM attached to the host.
2. Provide it in the ISO through a specified secondary device.
3. Serving it from an HTTP server.

The syntax to pass the configuration file to the kernel through the CD-ROM takes the following form:

ks=cdrom:/<config_file_path>

For example:

ks=cdrom:/isolinux/ks.cfg

The syntax to pass the configuration file to the kernel through a secondary device takes the following form:

ks=<device-path>:<path-referential-to-device>

For example:

ks=/dev/sr1:/isolinux/sample_ks.cfg

The syntax to serve the configuration file to the kernel from a HTTPS server takes the following form:

ks=https://<server>/<config_file_path>

To use HTTP path or self-signed HTTPS path, you have to enable insecure_installation by using insecure_installation=1 along with defining the ks path. The kernel command line argument, insecure_installation, acts as a flag that user can set to 1 to allow some operations that are not normally allowed due to security concerns. This is disabled by default and it is up to the user to the ensure security when this options is enabled.

HTTP example:

ks=http://<server>/<config_file_path> insecure_installation=1

HTTPS (self-signed) example:

ks=https://<server>/<config_file_path> insecure_installation=1

Building an ISO with a Kickstart Config File

Here’s an example of how to add a kickstart config file to the Photon OS ISO by mounting the ISO on an Ubuntu machine and then rebuilding the ISO. The following example assumes you can adapt the sample kickstart configuration file that comes with the Photon OS ISO to your needs. You can obtain the Photon OS ISO for free from VMware at the following URL:

https://packages.vmware.com/photon

Once you have the ISO, mount it.

mkdir /tmp/photon-iso
sudo mount photon.iso /tmp/photon-iso

Then copy the content of the ISO to a writable directory and push it into the directory stack:

mkdir /tmp/photon-ks-iso
cp -r /tmp/photon-iso/* /tmp/photon-ks-iso/
pushd /tmp/photon-ks-iso/

Next, copy the sample kickstart configuration file that comes with the Photon OS ISO and modify it to suit your needs. In the ISO, the sample kickstart config file appears in the isolinux directory and is named sample_ks.cfg. The name of the directory and the name of the file might be in all uppercase letters.

cp isolinux/sample_ks.cfg isolinux/my_ks.cfg
nano isolinux/my_ks.cfg

With a copy of the sample kickstart config file open in nano, make the changes that you want.

Now add a new item to the installation menu by modifying isolinux/menu.cfg and boot/grub2/grub.cfg:

cat >> isolinux/menu.cfg << EOF
label my_unattended
	menu label ^My Unattended Install
	menu default
	kernel vmlinuz
	append initrd=initrd.img root=/dev/ram0 ks=<ks_path>/my_ks.cfg loglevel=3 photon.media=cdrom
EOF


cat >> boot/grub2/grub.cfg << EOF
set default=0
set timeout=3
loadfont ascii
set gfxmode="1024x768"
gfxpayload=keep
set theme=/boot/grub2/themes/photon/theme.txt
terminal_output gfxterm
probe -s photondisk -u ($root)

menuentry "Install" {
    linux /isolinux/vmlinuz root=/dev/ram0 ks=<ks_path>/my_ks.cfg loglevel=3 photon.media=UUID=$photondisk
    initrd /isolinux/initrd.img
}
EOF 

Following is an example of the ks path:

`ks_path=cdrom:/isolinux`

Note: You can specify any mount media through which you want to boot Photon OS. To specify the mount media, specify the path of the mount media device in the photon.media field. You can specify the path as shown in the following syntax:

photon.media=/dev/<path of the Photon OS ISO>

Finally, rebuild the ISO so that it includes your kickstart config file:

mkisofs -R -l -L -D -b isolinux/isolinux.bin -c isolinux/boot.cat \
                -no-emul-boot -boot-load-size 4 -boot-info-table \
                -eltorito-alt-boot --eltorito-boot boot/grub2/efiboot.img -no-emul-boot \
                -V "PHOTON_$(date +%Y%m%d)" . > <new_iso_path>.iso


popd

3 - Kubernetes on Photon OS

You can use Kubernetes with Photon OS. The instructions in this section present a manual configuration that gets one worker node running to help you understand the underlying packages, services, ports, and so forth.

The Kubernetes package provides several services: kube-apiserver, kube-scheduler, kube-controller-manager, kubelet, kube-proxy. These services are managed by systemd. Their configuration resides in a central location: /etc/kubernetes.

3.1 - Prerequisites

You need two or more machines with version 3.0 “GA” or later of Photon OS installed. It is recommended to use the latest 4.0 GA version.

3.2 - Running Kubernetes on Photon OS

The procedure describes how to break the services up between the hosts.

The first host, photon-master, is the Kubernetes master. This host runs the kube-apiserver, kube-controller-manager, and kube-scheduler. In addition, the master also runs etcd. Although etcd is not needed on the master if etcd runs on a different host, this guide assumes that etcd and the Kubernetes master run on the same host. The remaining host, photon-node, is the node and runs kubelet, proxy, and docker.

3.2.1 - System Information

Hosts:

photon-master = 192.168.121.9
photon-node = 192.168.121.65

3.2.2 - Prepare the Hosts

The following packages have to be installed. If the tdnf command returns “Nothing to do,” the package is already installed.

• Install Kubernetes on all hosts (both photon-master and photon-node).

  tdnf install kubernetes
  

• Install iptables on photon-master and photon-node:

  tdnf install iptables
  

• Open the tcp port 8080 (api service) on the photon-master in the firewall

  iptables -A INPUT -p tcp --dport 8080 -j ACCEPT
  

• Open the tcp port 10250 (api service) on the photon-node in the firewall

  iptables -A INPUT -p tcp --dport 10250 -j ACCEPT
  

• Install Docker on photon-node:

  tdnf install docker
  

• Add master and node to /etc/hosts on all machines (not needed if the hostnames are already in DNS). Make sure that communication works between photon-master and photon-node by using a utility such as ping.

  echo "192.168.121.9	photon-master
  192.168.121.65	photon-node" >> /etc/hosts
  

• Edit /etc/kubernetes/config, which will be the same on all the hosts (master and node), so that it contains the following lines:

  # Comma separated list of nodes in the etcd cluster
  KUBE_MASTER="--master=http://photon-master:8080"
  
  # logging to stderr routes it to the systemd journal
  KUBE_LOGTOSTDERR="--logtostderr=true"
  
  # journal message level, 0 is debug
  KUBE_LOG_LEVEL="--v=0"
  
  # Should this cluster be allowed to run privileged docker containers
  KUBE_ALLOW_PRIV="--allow_privileged=false"
  

3.2.3 - Configure Kubernetes Services on the Master

Perform the following steps to configure Kubernetes services on the master:

1. Edit /etc/kubernetes/apiserver to appear as such. The service_cluster_ip_range IP addresses must be an unused block of addresses, not used anywhere else. They do not need to be routed or assigned to anything.

   # The address on the local server to listen to.
   KUBE_API_ADDRESS="--address=0.0.0.0"
   
   # Comma separated list of nodes in the etcd cluster
   KUBE_ETCD_SERVERS="--etcd-servers=http://127.0.0.1:4001"
   
   # Address range to use for services
   KUBE_SERVICE_ADDRESSES="--service-cluster-ip-range=10.254.0.0/16"
   
   # Add your own
   KUBE_API_ARGS=""
   

2. Start the appropriate services on master:

   for SERVICES in etcd kube-apiserver kube-controller-manager kube-scheduler; do
   	systemctl restart $SERVICES
   	systemctl enable $SERVICES
   	systemctl status $SERVICES
   done
   

3. To add the other node, create the following node.json file on the Kubernetes master node:

   {
       "apiVersion": "v1",
       "kind": "Node",
       "metadata": {
           "name": "photon-node",
           "labels":{ "name": "photon-node-label"}
       },
       "spec": {
           "externalID": "photon-node"
       }
   }
   

4. Create a node object internally in your Kubernetes cluster by running the following command:

   $ kubectl create -f ./node.json
   
   $ kubectl get nodes
   NAME                LABELS              STATUS
   photon-node         name=photon-node-label     Unknown
   

Note: The above example only creates a representation for the node photon-node internally. It does not provision the actual photon-node. Also, it is assumed that photon-node (as specified in name) can be resolved and is reachable from the Kubernetes master node.

3.2.4 - Configure the Kubernetes services on Node

Perform the following steps to configure the kubelet on the node:

1. Edit /etc/kubernetes/kubelet to appear like this:

   ###
   # Kubernetes kubelet (node) config
   
   # The address for the info server to serve on (set to 0.0.0.0 or "" for all interfaces)
   KUBELET_ADDRESS="--address=0.0.0.0"
   
   # You may leave this blank to use the actual hostname
   KUBELET_HOSTNAME="--hostname_override=photon-node"
   
   # location of the api-server
   KUBELET_API_SERVER="--kubeconfig=/etc/kubernetes/kubeconfig"
   
   # Add your own
   #KUBELET_ARGS=""
   

2. Make sure that the api-server end-point located in /etc/kubernetes/kubeconfig, targets the api-server in the master node and does not fall into the loopback interface:

   apiVersion: v1
   clusters:
   - cluster:
       server: <ip_master_node>:8080
   

3. Start the appropriate services on the node (photon-node):

   for SERVICES in kube-proxy kubelet docker; do 
       systemctl restart $SERVICES
       systemctl enable $SERVICES
       systemctl status $SERVICES 
   done
   

4. Check to make sure that the cluster can now see the photon-node on photon-master and that its status changes to Ready.

   kubectl get nodes
   NAME                LABELS              STATUS
   photon-node          name=photon-node-label     Ready
   

   If the node status is NotReady, verify that the firewall rules are permissive for Kubernetes.

   • Deletion of nodes: To delete photon-node from your Kubernetes cluster, one should run the following on photon-master (please do not do it, it is just for information):

   kubectl delete -f ./node.json
   

Result

You should have a functional cluster. You can now launch a test pod. For an introduction to working with Kubernetes, see Kubernetes documentation.

4 - Photon NFS Utilities for Mounting Remote File Systems

This document describes how to mount a remote file system on Photon OS by using nfs-utils, a commonly used package that contains tools to work with the Network File System protocol (NFS).

Check a Remote Server

showmount  -e nfs-servername or ip

Example:

showmount -e eastern-filer.eng.vmware.com
showmount -e 10.109.87.129

Mount a Remote File System in Photon Full

The nfs-utils package is installed by default in the full version of Photon OS. Here is how to mount a directory through NFS on Photon OS:

mount -t nfs nfs-ServernameOrIp:/exportfolder /mnt/folder

Example:

mount -t nfs eastern-filer.eng.vmware.com:/export/filer /mnt/filer
mount -t nfs 10.109.87.129:/export /mnt/export

Mount a Remote File System in Photon Minimal

The nfs-utils package is not installed in the minimal version of Photon OS. You install it by running the following command:

tdnf install nfs-utils

For more information on installing packages with the tdnf command, see the Photon OS Administration Guide.

Once nfs-utils is installed, you can mount a file system by running the following commands, replacing the placeholders with the path of the directory that you want to mount:

mount nfs
mount -t nfs nfs-ServernameOrIp:/exportfolder /mnt/folder