## Deprecation Notice

CSE Server and Kubernetes Container Clusters plugin will soon drop support for TKGi (previously known as Enterprise PKS). Consider using VMware Tanzu Kubernetes Grid (TKG) or VMware Tanzu Kubernetes Grid Service (TKGs) for management of Kubernetes clusters with VCD.

Enterprise PKS enablement

Overview

CSE 2.0 enables orchestration of K8 cluster deployments on VMware Enterprise PKS. At the same time, it maintains the CSE 1.x feature set of Native K8 cluster deployments directly on VMware Cloud Director. As a result, the capabilities of CSE 2.0 allow tenants to leverage both K8 Providers, Native and Enterprise PKS, for seamless K8 cluster deployments while ensuring clusters’ isolation between tenants. It also offers great flexibility to administrators to onboard tenants on K8 Provider(s) of their choice, be it Native and/or Enterprise PKS.

conceptual-view-cse

This page talks in detail about CSE 2.0 architecture with Enterprise PKS, the infrastructure set-up, configuration steps, as well as, key command line interfaces for K8 deployments.

Architecture

CSE 2.0 architecture comprises of Enterprise PKS Infrastructure stack, VMware Cloud Director Infrastructure stack, and CSE 2.0 modules. The Enterprise PKS Infrastructure stack is necessary only if there is an intention to leverage it for K8 cluster deployments. The diagram below illustrates a physical view of the complete infrastructure, as well as, its logical mapping in to VMware Cloud Director hierarchy, for ease of understanding.

Legend:

provider-setup

Infrastructure set-up and configuration

Before you begin

  1. Ensure fresh installation of Enterprise PKS infrastructure stack. Also, ensure there are no prior K8 cluster deployments on this stack.
  2. Ensure CSE, vCloud Director infrastructure stack, and Enterprise PKS infrastructure stack are all in the same management network, without proxy in between.

Enterprise PKS on-boarding

Below timeline diagram depicts infrastructure set-up and tenant on-boarding. Cloud-provider has to do below steps before on-boarding tenants.

  1. Set up one or more Enterprise PKS-vSphere-NSX-T instances.
  2. Ensure OpenID Connect feature is disabled on each Enterprise-PKS instance. Refer FAQ for more details.
  3. Create Enterprise PKS service accounts per each Enterprise PKS instance.
  4. On-board Enterprise PKS instance(s) in VCD
    • Attach Enterprise PKS’ corresponding vSphere in VCD through VCD UI.
    • Create provider-vdc(s) in VCD from underlying resources of newly attached Enterprise PKS’ vSphere(s). Ensure these pvdc(s) are dedicated for Enterprise PKS K8 deployments only.
  5. Install, configure and start CSE
    • Follow instructions to install CSE 2.0 beta here
    • Use cse sample command to generate config.yaml and pks.yaml skeleton config files.
    • Configure config.yaml with VCD details.
    • Configure pks.yaml with Enterprise PKS details. This file is necessary only if there is an intention to leverage Enterprise PKS for K8 deployments. Refer here for more details on how to fill in pks.yaml.
    • Run cse install command. Specify the Enterprise PKS configuration file along with regular CSE configuration file via the flag –pks-config-file and –config respectively. The install process will prepare NSX-T(s) of Enterprise PKS instances for tenant isolation. Ensure this command is run again for on-boarding of new Enterprise PKS instances at later point of time.
    • Start the CSE service. Specify the Enterprise PKS configuration file along with regular CSE configuration file via the flag –pks-config-file and –config respectively.

Enabling Enterprise PKS as a K8s provider changes the default behavior of CSE as described below. Presence of option --pks-config <pks-config-file> while executing cse run gives an indication to CSE that Enterprise PKS is enabled (in addition to Native VCD) as a K8s provider in the system.

If CSE runs without --pks-config-file option, there will not be any change in CSE’s default behavior i.e., all ovdc-s are open for native K8s cluster deployments.

Tenant on-boarding

  1. Create ovdc(s) in tenant organization from newly created provider-vdc(s) above via VCD UI. Do not choose Pay-as-you-go model for ovdc(s). Refer FAQ for more details.
  2. Use these CSE commands to grant K8 deployment rights to chosen tenants and tenant-users. Refer RBAC feature for more details
  3. Use CSE command to enable organization vdc(s) with a chosen K8-provider (native (or) TKGi).

Below diagram illustrates a time sequence view of setting up the infrastructure for CSE 2.0, followed by the on boarding of tenants. The expected steps are executed by Cloud providers or administrators.

provider-setup

CSE, VCD, Enterprise PKS Component Illustration

Below diagram outlines the communication flow between components for the tenant’s work-flow to create a new K8 cluster.

Legend:

Refer tenant-work-flow to understand the below decision box in grey color in detail. communication-flow

Tenant work-flow of create-cluster operation

To understand the creation of new K8 cluster work-flow in detail, review below flow chart in its entirety. In this illustration, user from tenant “Pepsi” attempts to create a new K8 cluster in organization VDC “ovdc-1”, and based on the administrator’s enablement for “ovdc-1”, the course of action can alter. tenant-work-flow

CSE commands

Administrator commands to on board a tenant

Granting rights to Tenants and Users:

Below steps of granting rights are required only if RBAC feature is turned on.

* vcd right add "{cse}:CSE NATIVE DEPLOY RIGHT" -o tenant1
* vcd right add "{cse}:CSE NATIVE DEPLOY RIGHT" -o tenant2
* vcd right add "{cse}:PKS DEPLOY RIGHT" -o tenant1

* vcd role add-right "Native K8 Author" "{cse}:CSE NATIVE DEPLOY RIGHT"
* vcd role add-right "PKS K8 Author" "{cse}:PKS DEPLOY RIGHT"
* vcd role add-right "Omni K8 Author" "{cse}:CSE NATIVE DEPLOY RIGHT"
* vcd role add-right "Omni K8 Author" "{cse}:PKS DEPLOY RIGHT"

* vcd user create 'native-user' 'password' 'Native K8 Author'
* vcd user create 'pks-user' 'password' 'PKS K8 Author'
* vcd user create 'power-user' 'password' 'Omni K8 Author'

Enabling ovdc(s) for TKGi deployments: Starting CSE 3.0, separate command group has been dedicated to TKGi (Enterprise PKS)

* vcd cse pks ovdc list
* vcd cse pks ovdc enable ovdc2 -o tenant1 -k ent-pks --pks-plan "gold" --pks-cluster-domain "tenant1.com"

Cluster management commands

Starting CSE 3.0, separate command group has been dedicated to TKGi (Enterprise PKS)

* vcd cse pks cluster list
* vcd cse pks cluster create
* vcd cse pks cluster info
* vcd cse pks cluster resize
* vcd cse pks cluster delete

FAQ

Enterprise PKS Limitations