Enterprise PKS enablement

Overview

CSE 2.0 enables orchestration of K8 cluster deployments on VMware Enterprise PKS. At the same time, it maintains the CSE 1.x feature set of Native K8 cluster deployments directly on VMware vCloud Director. As a result, the capabilities of CSE 2.0 allow tenants to leverage both K8 Providers, Native and Enterprise PKS, for seamless K8 cluster deployments while ensuring clusters’ isolation between tenants. It also offers great flexibility to administrators to onboard tenants on K8 Provider(s) of their choice, be it Native and/or Enterprise PKS.

conceptual-view-cse

This page talks in detail about CSE 2.0 architecture with Enterprise PKS, the infrastructure set-up, configuration steps, as well as, key command line interfaces for K8 deployments.

Architecture

CSE 2.0 architecture comprises of Enterprise PKS Infrastructure stack, vCloud Director Infrastructure stack, and CSE 2.0 modules. The Enterprise PKS Infrastructure stack is necessary only if there is an intention to leverage it for K8 cluster deployments. The diagram below illustrates a physical view of the complete infrastructure, as well as, its logical mapping in to vCloud Director hierarchy, for ease of understanding.

Legend:

provider-setup

Infrastructure set-up and configuration

Before you begin

  1. Ensure fresh installation of Enterprise PKS infrastructure stack. Also, ensure there are no prior K8 cluster deployments on this stack.
  2. Ensure CSE, vCloud Director infrastructure stack, and Enterprise PKS infrastructure stack are all in the same management network, without proxy in between.

Enterprise PKS on-boarding

Below timeline diagram depicts infrastructure set-up and tenant on-boarding. Cloud-provider has to do below steps before on-boarding tenants.

  1. Set up one or more Enterprise PKS-vSphere-NSX-T instances.
  2. Ensure OpenID Connect feature is disabled on each Enterprise-PKS instance. Refer FAQ for more details.
  3. Create Enterprise PKS service accounts per each Enterprise PKS instance.
  4. On-board Enterprise PKS instance(s) in vCD
    • Attach Enterprise PKS’ corresponding vSphere in vCD through vCD UI.
    • Create provider-vdc(s) in vCD from underlying resources of newly attached Enterprise PKS’ vSphere(s). Ensure these pvdc(s) are dedicated for Enterprise PKS K8 deployments only.
  5. Install, configure and start CSE
    • Follow instructions to install CSE 2.0 beta here
    • Use cse sample command to generate config.yaml and pks.yaml template files.
    • Configure config.yaml with vCD and K8 template details.
    • Configure pks.yaml with Enterprise PKS details. This file is necessary only if there is an intention to leverage Enterprise PKS for K8 deployments. Refer pks_config key in config.yaml for more details on how to fill in pks.yaml and how it’s presence changes the CSE’s default behavior on user’s ability to deploy (Native/Enterprise PKS) K8 clusters in any given organization vdc.
    • Run CSE install command. It prepares NSX-T(s) of Enterprise PKS instances for tenant isolation. Ensure this command is run for on-boarding of new Enterprise PKS instances at later point of time.
    • Start the CSE service.

Tenant on-boarding

  1. Create ovdc(s) in tenant organization from newly created provider-vdc(s) above via vCD UI. Do not choose Pay-as-you-go model for ovdc(s). Refer FAQ for more details.
  2. Use these CSE commands to grant K8 deployment rights to chosen tenants and tenant-users. Refer RBAC feature for more details
  3. Use CSE command to enable organiation vdc(s) with a chosen K8-provider (native (or) ent-pks).

Below diagram illustrates a time sequence view of setting up the infrastructure for CSE 2.0, followed by the on boarding of tenants. The expected steps are executed by Cloud providers or administrators.

provider-setup

CSE, vCD, Enterprise PKS Component Illustration

Below diagram outlines the communication flow between components for the tenant’s workflow to create a new K8 cluster.

Legend:

Refer tenant-workflow to understand the below decision box in grey color in detail. communication-flow

Tenant workflow of create-cluster operation

To understand the creation of new K8 cluster workflow in detail, review below flow chart in its entirety. In this illustration, user from tenant “Pepsi” attempts to create a new K8 cluster in organization VDC “ovdc-1”, and based on the administrator’s enablement for “ovdc-1”, the course of action can alter. tenant-workflow

CSE commands

Administrator commands to on board a tenant

Granting rights to Tenants and Users:

Below steps of granting rights are required only if RBAC feature is turned on.

* vcd right add "{cse}:CSE NATIVE DEPLOY RIGHT" -o tenant1
* vcd right add "{cse}:CSE NATIVE DEPLOY RIGHT" -o tenant2
* vcd right add "{cse}:PKS DEPLOY RIGHT" -o tenant1
* vcd role add-right "Native K8 Author" "{cse}:CSE NATIVE DEPLOY RIGHT"
* vcd role add-right "PKS K8 Author" "{cse}:PKS DEPLOY RIGHT"
* vcd role add-right "Omni K8 Author" "{cse}:CSE NATIVE DEPLOY RIGHT"
* vcd role add-right "Omni K8 Author" "{cse}:PKS DEPLOY RIGHT"
* vcd user create 'native-user' 'password' 'Native K8 Author'
* vcd user create 'pks-user' 'password' 'PKS K8 Author'
* vcd user create 'power-user' 'password' 'Omni K8 Author'

Enabling ovdc(s) for a particular K8-provider:

* vcd cse ovdc list
* vcd cse ovdc enable ovdc1 -o tenant1 -k native
* vcd cse ovdc enable ovdc2 -o tenant1 -k ent-pks --pks-plan "gold" --pks-cluster-domain "tenant1.com"
* vcd cse ovdc enable ovdc1 -o tenant2 -k native

Cluster management commands

* vcd cse cluster list
* vcd cse cluster create
* vcd cse cluster info
* vcd cse cluster resize
* vcd cse cluster delete

FAQ

Compatibility matrix

CSE Supported vCD Versions Enterprise PKS NSX-T
2.0 Beta 9.5, 9.7 1.4 2.3
2.0.0 9.5, 9.7 1.4 2.3
2.0.0 9.5, 9.7 1.4 2.4