Container Service Extension (CSE) is a VMware vCloud Director extension that helps tenants create and work with Kubernetes clusters.

Table of Contents

(Section titles link back to table of contents)



Overview

CSE brings Kubernetes as a service to vCD by creating customized VM templates and enabling tenant/organization administrators to deploy fully functional Kubernetes clusters.

System/cloud administrators handle setting up vCD, CSE config file, CSE Server, and VM templates.

Once CSE Server is running and VM templates are available, tenant/organization administrators and users can use CSE client (vcd-cli) to handle Kubernetes cluster management (deploying clusters, adding worker nodes/NFS nodes/disks/etc)

cse-overview cse-roles



Getting CSE

Install Python 3.6 or greater. Pip, Python’s package manager, should come with Python.

Verify python and pip installation:

> python3 --version
Python 3.7.0

> pip3 --version
pip 18.0 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)


Install and verify CSE:

> pip3 install container-service-extension
...

> cse version
CSE, Container Service Extension for VMware vCloud Director, version 1.2.0


Alternatively, a specific version of CSE can be installed from GitHub

> pip3 install git+https://github.com/vmware/container-service-extension.git@1.2.0

All CSE versions on GitHub: https://github.com/vmware/container-service-extension/tags



Enabling CSE Client Commands in vcd-cli

CSE client is not enabled yet:

> vcd cse version
Error: No such command "cse".


Edit ~/.vcd-cli/profiles.yaml to include this section:

extensions:
- container_service_extension.client.cse


If ~/.vcd-cli/profiles.yaml doesn’t exist, logging in to vCD via vcd-cli will create it

> vcd login IP ORGNAME USERNAME -iwp PASSWORD


Verify that CSE client works:

> vcd cse version
CSE, Container Service Extension for VMware vCloud Director, version 1.2.0


System/Cloud Administrator

CSE/vCD Compatibility

| CSE version | vCD version | |————-|———————| | 1.1.0 | 8.20, 9.0, 9.1 | | 1.2.0 | 8.20, 9.0, 9.1, 9.5 |


Config File

Generate a skeleton config yaml file:

> cse sample > config.yaml

Edit this file with values from your vCloud Director


Config file has 5 sections: amqp, vcd, vcs, service, broker

amqp section

CSE Server will communicate with vCD using these settings

During CSE Server installation, CSE can configure vCD’s AMQP settings to match these settings


vcd section

It’s recommended to create a service account for vCD with minimum required privileges. An attacker getting credentials for a user account with admin-level privileges can be catastrophic

Minimum set of rights required by a service account to install CSE properly (subject to change with new vCD versions):

Always ensure vCD service account has enough privileges. Alternatively, you can create a role with Admin privileges and deselect/delete rights which are not required.


vcd section has the following properties:

Property Value
host IP or hostname of the vCloud Director
username Username of the vCD service account with minimum roles and rights
password Password of the vCD service account.


vcs section

Guest Operations Privileges required for vCenter service account:


Each vc under vcs section has the following properties:

Property Value
name Name of the vCenter registered in vCD
username Username of the vCenter service account with minimum of guest-operation privileges.
password Password of the vCenter service account.


service section

Specify how many threads you want CSE Server to create.


broker section

broker section has the following properties:

Property Value
type Broker type, set to default
org vCD organization that contains the shared catalog where the master templates will be stored
vdc Virtual datacenter within org that will be used during the install process to build the template
network Org Network within vdc that will be used during the install process to build the template. It should have outbound access to the public Internet. The CSE appliance doesn’t need to be connected to this network
ip_allocation_mode IP allocation mode to be used during the install process to build the template. Possible values are dhcp or pool. During creation of clusters for tenants, pool IP allocation mode is always used
catalog Public shared catalog within org where the template will be published
cse_msg_dir Reserved for future use
storage_profile Name of the storage profile to use when creating the temporary vApp used to build the template
default_template Name of the default template to use if none is specified
templates A list of templates available for clusters

Each template in the templates property has the following properties:

Property Value
name Unique name of the template
source_ova URL of the source OVA to download
sha256_ova sha256 of the source OVA
source_ova_name Name of the source OVA in the catalog
catalog_item Name of the template in the catalog
description Information about the template
temp_vapp Name of the temporary vApp used to build the template. Once the template is created, this vApp can be deleted. It will be deleted by default during the installation based on the value of the cleanup property
cleanup If True, temp_vapp will be deleted by the install process after the master template is created
admin_password root password for the template and instantiated VMs. This password should not be shared with tenants
cpu Number of virtual CPUs to be allocated for each VM
mem Memory in MB to be allocated for each VM

VM Templates

CSE supports multiple VM templates to create Kubernetes clusters from. Templates may vary in guest OS or software versions, and must have a unique name. One template must be defined as the default template, and tenants have the option to specify the template to use during cluster/node creation.

Source .ova Files for VM Templates

OS OVA Name URL SHA256
Photon OS 1.0, Rev 2 photon-custom-hw11-1.0-62c543d.ova https://bintray.com/vmware/photon/download_file?file_path=photon-custom-hw11-1.0-62c543d.ova 6d6024c5531f5554bb0d2f51f3005078ce6d4ee63c142f2453a416824c5344ca
Photon OS 2.0 GA photon-custom-hw11-2.0-304b817.ova http://dl.bintray.com/vmware/photon/2.0/GA/ova/photon-custom-hw11-2.0-304b817.ova cb51e4b6d899c3588f961e73282709a0d054bb421787e140a1d80c24d4fd89e1
Ubuntu 16.04.4 LTS ubuntu-16.04-server-cloudimg-amd64.ova https://cloud-images.ubuntu.com/releases/xenial/release-20180418/ubuntu-16.04-server-cloudimg-amd64.ova 3c1bec8e2770af5b9b0462e20b7b24633666feedff43c099a6fb1330fcc869a9


Updating VM Templates

CSE Server should be gracefully stopped before updating VM templates, to avoid errors that can occur when using vcd cse cluster create ... or vcd cse node create ...

In general, updating a template doesn’t have any effect on existing Kubernetes master and worker nodes. CSE and template compatibility can be found in release notes.

Templates can also be generated on a vCD instance that CSE Server is not registered to. Templates can be generated in multiple vCD instances in parallel.


Update a template:

> cse install -c config.yaml --template photon-v2 --update --amqp skip --ext skip


Updating a template increases versionNumber of the corresponding catalog item by 1:

> vcd catalog info cse photon-custom-hw11-2.0-304b817-k8s

CSE Server

Installing CSE Server

CSE Server should be installed by the vCloud Director System/Cloud Administrator on a new VM or one of the existing servers that are part of vCD. This CSE VM is the CSE appliance.

CSE appliance requires network access to the vCD cell, vCenter(s), and AMQP server.

CSE appliance does not require access to the network(s) where the Kubernetes templates will be created (network and temp_vapp config file properties) or the tenant network(s) where the clusters will be created.

cse install command supports the following options:

Option Short Argument(s) Description Default Value
–config -c path/to/config.yaml Config file to use config.yaml
–template -t template-name Install the specified template ‘*’ (installs all templates specified in config file)
–update -u n/a Recreate templates during installation False
–no-capture -n n/a Don’t capture the temporary vApp as a template (Leaves it standing for debugging purposes) False
–ssh-key -k path/to/ssh-key.pub ssh-key file to use for vm access (root password ssh access is disabled for security reasons) None
–amqp -a prompt OR skip OR config prompt: ask before configuring AMQP settings
skip: do not configure AMQP settings
config: configure AMQP without asking for confirmation
prompt
–ext -e prompt OR skip OR config prompt: ask before registering CSE
skip: do not register CSE
config: register CSE without asking for confirmation
prompt


To monitor the vApp customization process, you can ssh into the temporary vApp. In the temporary vApp, the output of the customization script is captured in /tmp/FILENAME.out as well as /tmp/FILENAME.err:

# print out file contents as it's being written to
> tail -f /tmp/FILENAME.out
> tail -f /tmp/FILENAME.err

The temporary vApp guest OS does not allow root ssh access via password for security reasons (use --ssh-key option to provide a public key).

To inspect the temporary vApp after customization, use the --no-capture option (also requires the --ssh-key option):

> cse install -c config.yaml --no-capture --ssh-key ~/.ssh/id_rsa.pub


cse-install

Validate CSE Installation

Validate that CSE has installed correctly with:

> cse check --config config.yaml --check-install

The cse check command supports the following options:

Option Short Argument(s) Description Default
–config -c path/to/config.yaml Config file to use config.yaml
–check-install -i n/a Check CSE installation on vCD False
–template -t template-name If --check-install is set, check that the specified template exists ‘*’ (checks for all templates specified in config file)


Validate that CSE has been registered in vCD Using vcd-cli, check that the extension has been registered in vCD:

# login as system administrator
> vcd login vcd.serviceprovider.com System administrator --password passw0rd -w -i

# list extensions
> vcd system extension list

# get details of CSE extension
> vcd system extension info cse


Optional

Configure the API extension timeout (seconds) on the vCloud Director cell:

> cd /opt/vmware/vcloud-director/bin
> ./cell-management-tool manage-config -n extensibility.timeout -l
> ./cell-management-tool manage-config -n extensibility.timeout -v 20


Manually register CSE api extension to vCD:

> vcd system extension create cse cse cse vcdext '/api/cse, /api/cse/.*, /api/cse/.*/.*'

Operating CSE Server

CSE Server uses threads to process requests. The number of AMQP listener threads can be configured in the config file with listeners property under service section (default is 5).

Running CSE Server Manually

> cse run --config config.yaml

# run server in the background
> nohup cse run --config config.yaml > nohup.out 2>&1 &

Server output log can be found in cse.log


Running CSE Server as a Service

A sample systemd unit is provided by CSE. Copy file cse.service from where CSE is installed to, and move it to /etc/systemd/system/cse.service. A sample cse.sh is also provided. (what does cse.sh does and is it used in this?)

CSE service daemon should be started using systemctl start cse. To enable, disable, and stop CSE service, use CSE client.

# hook CSE unit into relevant places to automatically do things
# depending on what's specified in the unit file
> vcd cse system enable

# start CSE service now
> systemctl start cse

# stop processing new requests, and finish processing existing requests
# disables CSE service
> vcd cse system disable
property    value
----------  -------
message     Updated

# wait until all active threads have finished, then exits CSE service
> vcd cse system stop -y
property    value
----------  ---------------------------------------------------------------------
message     CSE graceful shutdown started. CSE will finish processing 4 requests.

> vcd cse system info
property              value
--------------------  ------------------------------------------------------
all_threads           8
config_file           /Users/pgomez/vmware/cse/testbed-202-34.yaml
consumer_threads      5
description           Container Service Extension for VMware vCloud Director
product               CSE
python                3.6.4
requests_in_progress  4
status                Shutting down
version               1.2.0


If CSE Server is disabled, users will get the following message when executing any CSE command:

$ vcd cse cluster list
Usage: vcd cse cluster list [OPTIONS]

Error: CSE service is disabled. Contact the System Administrator.


Monitoring CSE Service

vCD System Administrator can monitor CSE service status via CSE client:

> vcd cse system info
property              value
--------------------  ------------------------------------------------------
all_threads           10
config_file           /opt/vmware/cse/testbed-202-34.yaml
consumer_threads      6
description           Container Service Extension for VMware vCloud Director
product               CSE
python                3.6.4
requests_in_progress  3
status                Running
version               1.2.0


On Photon OS, to keep the service running after logout, check /etc/systemd/logind.conf and set KillUserProcesses to no

[Login]
KillUserProcesses=no


System administrators can list all the clusters running in vCD with a search command using cluster vApp metadata:

> vcd search adminvapp -f 'metadata:cse.cluster.id!=STRING:'

Maintaining CSE Server

When upgrading CSE versions, re-register the extension:

# remove previous registration of CSE
> vcd system extension delete cse

# run cse installation again
> cse install --config config.yaml


Updating CSE Server

Uninstalling CSE Server


Useful Commands

cse ... commands are used by system administrators to:

vcd cse ... commands are used by system administrators to:

### Use '-h' option to see help page and options for any command
> cse install --config config.yaml
> cse check --config config.yaml
> cse run --config config.yaml

# login to vCD to use vcd-cli commands
> vcd login IP system USERNAME -iwp PASSWORD

# set ORGNAME to be active org for this session
> vcd org use ORGNAME

# set VDCNAME to be active vdc for this session
> vcd vdc use VDCNAME


Tenant/Organization Administrator and Other Users

Once CSE Server is installed and running, tenants can use CSE client (vcd-cli) to create Kubernetes clusters on demand.

Master and worker nodes are VMs within a vApp on the tenant virtual datacenter. The VMs are required to be connected to an organization VDC network specified with the --network required command-line option. This network should have a static IP pool, and may require access to the internet if installing additional components.


Useful Commands

vcd cse ... commands are used by tenant/org administrators and users to:


Summary of commands available to manage templates, clusters and nodes:

Command Description
vcd cse template list List available templates to create clusters
vcd cse cluster create CLUSTER_NAME Create a new Kubernetes cluster
vcd cse cluster create CLUSTER_NAME --enable-nfs Create a new Kubernetes cluster with NFS PV support.
vcd cse cluster list List created clusters.
vcd cse cluster delete CLUSTER_NAME Delete a Kubernetes cluster.
vcd cse node create CLUSTER_NAME --nodes n Add n nodes to a cluster.
vcd cse node create CLUSTER_NAME --type nfsd Add an NFS node to a cluster.
vcd cse node list CLUSTER_NAME List nodes of a cluster.
vcd cse node delete CLUSTER_NAME NODE_NAME Delete nodes from a cluster.


By default, CSE Client will display the task progress until the task finishes or fails. The --no-wait will return the task information, which you can use to monitor the task progress:

> vcd --no-wait cse cluster create CLUSTER_NAME --network intranet --ssh-key ~/.ssh/id_rsa.pub

# displays the status and progress of the task
> vcd task wait 377e802d-f278-44e8-9282-7ab822017cbd

# lists the current running tasks in the organization
> vcd task list running

Automation

CSE can be easily scripted to automate the creation and operation of Kubernetes clusters and nodes.

Users can also interact with CSE via the Python package or the CSE API exposed in vCD.

This Python script creates a Kubernetes cluster on vCloud Director:

#!/usr/bin/env python3
from pyvcloud.vcd.client import BasicLoginCredentials
from pyvcloud.vcd.client import Client
from container_service_extension.client.cluster import Cluster

client = Client('vcd.mysp.com')
client.set_credentials(BasicLoginCredentials('usr1', 'org1', '******'))

cse = Cluster(client)
result= cse.create_cluster('vdc1', 'net1', 'cluster1')
task = client.get_resource(result['task_href'])
task = client.get_task_monitor().wait_for_status(task)
print(task.get('status'))

client.logout()

Example Use Case

# create cluster mycluster with one master and two nodes, connected to provided network
# a public key is provided to be able to ssh into the VMs
> vcd cse cluster create mycluster --network intranet --ssh-key ~/.ssh/id_rsa.pub

# list the worker nodes of a cluster
> vcd cse node list mycluster

# create cluster mycluster with one master, three nodes and connected to provided network
> vcd cse cluster create mycluster --network intranet --nodes 3 --ssh-key ~/.ssh/id_rsa.pub

# create a single worker node cluster, connected to the specified network. Nodes can be added later
> vcd cse cluster create mycluster --network intranet --nodes 0 --ssh-key ~/.ssh/id_rsa.pub

# create cluster mycluster with one master, three worker nodes, connected to provided network
# and one node of type NFS server
> vcd cse cluster create mycluster --network intranet --nodes 3 --ssh-key ~/.ssh/id_rsa.pub --enable-nfs

# add 2 worker nodes to a cluster with 4GB of ram and 4 CPUs each, from the photon-v2 template
# and using the specified storage profile
> vcd cse node create mycluster --nodes 2 --network intranet --ssh-key ~/.ssh/id_rsa.pub --memory 4096 --cpu 4 --template photon-v2 --storage-profile Development

# add 1 nfsd node to a cluster with 4GB of ram and 4 CPUs each, from the photon-v2 template
# and using the specified storage profile
> vcd cse node create mycluster --nodes 1 --type nfsd --network intranet --ssh-key ~/.ssh/id_rsa.pub --memory 4096 --cpu 4 --template photon-v2 --storage-profile Development

# info on a given node. If the node is of type nfsd, it displays info about Exports.
> vcd cse node info mycluster nfsd-dj3s

# delete 2 nodes from a cluster
> vcd cse node delete mycluster node-dj3s node-dj3s --yes

# list available clusters
> vcd cse cluster list

# info on a given cluster
> vcd cse cluster info

# retrieve cluster config
> vcd cse cluster config mycluster > ~/.kube/config

# check cluster configuration
> kubectl get nodes

# deploy a sample application
> kubectl create namespace sock-shop

> kubectl apply -n sock-shop -f "https://github.com/microservices-demo/microservices-demo/blob/master/deploy/kubernetes/complete-demo.yaml?raw=true"

# check that all pods are running and ready
> kubectl get pods --namespace sock-shop

# access the application
> IP=`vcd cse cluster list|grep '\ mycluster'|cut -f 1 -d ' '`
> open "http://${IP}:30001"

# delete cluster when no longer needed
> vcd cse cluster delete mycluster --yes


NFS-based Static Persistent Volumes

Link



cse-usage



Troubleshooting

cse.log logs CSE Server activity. Server requests and responses are recorded here, as well as outputs of scripts that were run on VMs.

cse-check.log logs CSE operations, such as cse install. Stack traces and HTTP messages specific to CSE are recorded here.

vcd.log logs vcd-cli and pyvcloud activity. Stack traces and HTTP messages specific to vcd-cli are recorded here.

Common mistakes:



Known Issues

Failures during template creation during installation


CSE service fails to start


CSE fails to connect to vSphere 6.7. Please follow below workaround while we are working on the fix.


CSE does not clean up after itself if something goes wrong. When CSE installation is aborted for any reason, ensure temporary vApp is deleted in vCD before re-issuing the install command


pip install container-service-extension while in a virtual environment will result in CSE not being able to find the script files it requires. Fix is in review.


During cse install, if vCD and config file AMQP configuration is the same, but the specified AMQP exchange has not been created yet, cse install will skip the exchange creation. Fix is in review.


Currently, NFS servers in Kubernetes cluster can not only be accessed by nodes of that cluster but also by any VM (outside of the cluster) residing in the same orgVdc. Ideal solution is to have vApp network created for each Kubernetes cluster, which is in our road-map to implement. Until then, please choose one of below workarounds to avert this problem if the need arises.



Contributing Guide

Link



Release Notes

Link