vcf-services

Runtime Spec and Overlay System

The Runtime Spec and Overlay system enables dynamic configuration transformation in VCF Services. It provides a mechanism for CRs to depend on values that are only known at reconciliation time - such as an IP address assigned by a Supervisor, a certificate generated by a workload, or a password resolved from a Secret - and for those values to be injected into other CRs via ytt-based transformations.

Key Concepts

Base spec = the original CR specification as authored in the bundle, containing selector references and placeholder values.

Runtime spec = the fully expanded spec produced during reconciliation; all selector-based fields have been resolved to their concrete values.

Overlay = a ytt-based transformation defined in an Overlay CR that is applied to a runtime spec before the CR’s operations are executed. See Overlay for the CR reference.

Runtime Spec

What It Contains

When a CR is reconciled, the controller resolves all selector-defined fields and builds a runtime spec. The runtime spec:

Contains the actual values resolved from referenced Secrets, ConfigMaps, and other CRs
Contains resolved package definitions and access grant configurations
Contains actual passwords, certificates, and credentials
Does not contain selector definitions or placeholder values

How Selectors Are Resolved

The following example shows a SupervisorService spec as authored and the runtime spec produced from it:

Authored spec (selectors unresolved):

apiVersion: services.vcfa.broadcom.com/v2
kind: SupervisorService
spec:
  region: us-west-1
  supervisor: supervisor-vc1
  values:
    selector:
      kind: Secret
      labels:
        services.vcfa.broadcom.com/context: install
      path: data.values
  package:
    selector:
      kind: ConfigMap
      labels:
        services.vcfa.broadcom.com/context: supervisor-service

Runtime spec (selectors resolved):

runtimeSpec:
  region: us-west-1
  supervisor: supervisor-vc1
  values:
    arcturusAdminPassword: Arcturus12345
    secretKey: 0123456789ABCDEF
    database:
      password: change-it
    persistence:
      persistentVolumeClaim:
        registry:
          storageClass: vks-content-library
          size: 10Gi
  package:
    apiVersion: data.packaging.carvel.dev/v1alpha1
    kind: Package
    metadata:
      name: arcturus.galaxy.com.2.11.1
    spec:
      refName: arcturus.galaxy.com
      version: 2.11.1+galaxy.1
  packageMetadata:
    apiVersion: data.packaging.carvel.dev/v1alpha1
    kind: PackageMetadata
    metadata:
      name: arcturus.galaxy.com
    spec:
      displayName: "Arcturus Registry"

Overlay Application

Once the runtime spec is built, the controller checks whether the CR carries an services.vcfa.broadcom.com/overlay label. If it does, the overlay transformation is applied to the runtime spec before the CR’s operations proceed. If no overlay label is present, the runtime spec is used as-is.

Base spec (with selectors)
        │
        │ 1. Resolve all selectors
        ▼
Runtime spec (concrete values)
        │
        │ 2. Overlay label present?
        ▼
   ┌────────┐
   │Overlay?│
   └───┬────┘
       │
   No  │  ──────────────► Use runtime spec as-is
       │
   Yes │
       │ 3. Fetch Overlay Secret (values + ytt template)
       ▼
Transformed runtime spec
       │
       │ 4. Execute CR operations
       ▼
Deploy / Configure target system

Overlay System

How Overlays Work

An Overlay CR declares:

A set of data values - each value is either a literal string or a selector that resolves a value from another resource in the cluster.
A ytt template - the transformation to apply to the runtime spec of any CR that references this Overlay.

When all data values are resolved, the controller writes them plus the ytt template into a Kubernetes Secret that shares the same name as the Overlay CR. CRs that reference the overlay by label fetch this secret and pass it through ytt to produce a transformed runtime spec.

Linking an Overlay to a Target CR

The link is established by a matching label value on both the Overlay CR and the target CR:

# Overlay CR
apiVersion: services.vcfa.broadcom.com/v2
kind: Overlay
metadata:
  labels:
    services.vcfa.broadcom.com/overlay: install.us-west-1

# Target CR (e.g. SupervisorService)
apiVersion: services.vcfa.broadcom.com/v2
kind: SupervisorService
metadata:
  labels:
    services.vcfa.broadcom.com/overlay: install.us-west-1
    services.vcfa.broadcom.com/overlay-apply: eager

The services.vcfa.broadcom.com/overlay label value must be identical on both resources.

Application Modes

The services.vcfa.broadcom.com/overlay-apply label on the target CR controls what happens when the Overlay is not yet ready:

`eager` (default)

The target CR blocks until the Overlay Secret is available and Healthy. While waiting, the CR reports status: Busy. Reconciliation is retried every 10 seconds.

Use eager when the overlay contains values that are critical for the CR to function correctly - the CR must not be deployed without them.

Status: Busy
Message: "Waiting for overlay: Overlay secret 'install.us-west-1' not found or not ready
          (mode: eager). The Overlay CR with label
          'services.vcfa.broadcom.com/overlay=install.us-west-1' must exist and have
          status=Healthy. Will retry in 10s."

`lazy`

The target CR proceeds with its base runtime spec if the Overlay is not yet ready. On each subsequent reconciliation, if the Overlay Secret is now available, the transformation is applied. No blocking occurs.

Use lazy when the overlay contains optional or non-critical configuration that can be applied after initial deployment.

# First reconciliation (overlay not yet ready)
Status: Healthy
Message: "Deployed with base configuration"

# After overlay becomes available
Status: Healthy
Message: "Overlay applied successfully"

Overlay Data Values

The spec.data field of an Overlay CR defines the values available to the ytt template under data.values.*.

Value Types

spec:
  data:
    # Literal value - used immediately
    registry_port:
      value: "5000"

    # Selector - resolved from another resource in the cluster
    depot_ip:
      selector:
        kind: SupervisorService
        labels:
          services.vcfa.broadcom.com/context: install
          services.vcfa.broadcom.com/link-region: us-west-1
        path: status.readyConditions.externalIp

    # Selector with escaped dot in key name
    tls_certificate:
      selector:
        kind: Secret
        labels:
          app: arcturus
        path: data.tls\.crt

Resolution Behaviour

The Overlay controller iterates over all spec.data entries on every reconciliation:

Literal values are used immediately.
Selector values are fetched from the cluster by matching the specified kind and labels, then extracting the field at path.
If any selector value is nil, empty, or the referenced resource does not yet exist, the Overlay reports status: Busy and requeues after 10 seconds.
Once all values are resolved, the controller writes a Kubernetes Secret containing the resolved values (JSON) and the ytt template.

Reconciliation Intervals

Overlay state	Reconciliation interval	Trigger
`Busy`	10 seconds	Waiting for selector values to become available
`Healthy`	~10 minutes	Periodic re-evaluation of selector values
Any	Immediate	Overlay CR spec or annotation change

Target CRs reconcile immediately when the Overlay Secret is created or updated (via Kubernetes watch).

ytt Overlay Templates

The spec.overlay field is a multi-line string containing a ytt overlay document. The resolved data values are available under data.values.*.

Basic structure

spec:
  overlay: |
    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"values": {"arcturus": {}}})
    ---
    values:
      arcturus:
        externalURL: #@ data.values.new_url
        adminPassword: #@ data.values.admin_password

Important: The ytt template operates on the runtime spec - the selector-resolved structure - not the original authored spec. Ensure your #@overlay/match patterns target the resolved field names and structure.

Matching strategies

# Match by exact structure subset
#@overlay/match by=overlay.subset({"kind": "Secret", "metadata": {"name": "app-config"}})

# Match by index within an array
#@overlay/match by=overlay.index(0)

# Match all items in an array
#@overlay/match by=overlay.all

# Optional match - succeed even if zero matches
#@overlay/match by=overlay.subset({"kind": "Service"}), expects="0+"

# Match with a custom function
#@overlay/match by=lambda i,left,right: left["name"] == "arcturus"

Complete example - inject dynamic IP into SupervisorService

apiVersion: services.vcfa.broadcom.com/v2
kind: Overlay
metadata:
  labels:
    services.vcfa.broadcom.com/overlay: publish.us-west-1
spec:
  data:
    backend_ip:
      selector:
        kind: SupervisorService
        labels:
          services.vcfa.broadcom.com/name: backend-service
          services.vcfa.broadcom.com/link-region: us-west-1
        path: status.readyConditions.externalIp
    tls_cert:
      selector:
        kind: SupervisorService
        labels:
          services.vcfa.broadcom.com/name: cert-manager
          services.vcfa.broadcom.com/link-region: us-west-1
        path: status.readyConditions.certificates
  overlay: |
    #@ load("@ytt:overlay", "overlay")

    #@overlay/match by=overlay.subset({"values": {}})
    ---
    values:
      selector:
        kind: Secret
        #@overlay/match-child-defaults missing_ok=True
        name: frontend-config
        path: data.values
      #@overlay/replace
      patch: |
        backend:
          host: #@ data.values.backend_ip
          port: 8080
          tls_enabled: #@ len(data.values.tls_cert) > 0
        security:
          certificate: #@ data.values.tls_cert if len(data.values.tls_cert) > 0 else ""

End-to-End Flow

The following trace shows how multiple SupervisorService CRs interact via an Overlay in a realistic scenario where a frontend service needs to consume both the external IP of a backend service and TLS certificates:

T+0s    Backend SupervisorService (producer) created with overlay label: install.us-west-1
         └─► SupervisorService: resolves selectors → runtime spec built
         └─► externalIp not yet available → ready condition pending

T+0s    Certificate Manager SupervisorService (cert-manager) created
         └─► SupervisorService: resolves selectors → runtime spec built
         └─► certificates not yet available → ready condition pending

T+0s    Overlay CR (publish.us-west-1) reconciles
         └─► backend_ip selector: queries Backend SupervisorService status.readyConditions.externalIp
         └─► tls_cert selector: queries Certificate Manager SupervisorService status.readyConditions.certificates
         └─► Values are nil → Overlay status: Busy, requeue in 10s

T+0s    Frontend SupervisorService (consumer) reconciles (overlay-apply: eager)
         └─► Overlay Secret not yet available → status: Busy, requeue in 10s

T+30s   Backend SupervisorService workload receives external IP: 10.0.1.100
         └─► status.readyConditions.externalIp = 10.0.1.100

T+45s   Certificate Manager SupervisorService generates certificates
         └─► status.readyConditions.certificates = "-----BEGIN CERTIFICATE-----..."

T+50s   Overlay reconciles (Busy → retries every 10s)
         └─► backend_ip resolved: 10.0.1.100
         └─► tls_cert resolved: certificate content
         └─► All values resolved → Overlay Secret created
         └─► Overlay status: Healthy

T+51s   Frontend SupervisorService watch fires on Secret creation
         └─► Fetches Overlay Secret (values + ytt template)
         └─► Applies ytt transformation to runtime spec
         └─► Transformed spec: updated with backend IP and TLS certificate

T+52s   Frontend SupervisorService applies transformed spec
         └─► Frontend service configuration updated with backend connection and TLS settings
         └─► status: Healthy

Value change propagation latency

Overlay state when source value changes	Maximum propagation latency
`Busy` (already polling)	~10 seconds
`Healthy` (periodic reconciliation)	~10 minutes

The target CR reconciles immediately after the Overlay Secret is updated (via Kubernetes watch), so the dominant factor is the Overlay’s reconciliation interval.

Troubleshooting

Overlay stuck in `Busy` - selector values not resolving

Symptom:

Status: Busy
Message: "Waiting for values: [depot_ip]"

Likely causes:

The referenced resource does not exist yet or has not reached a Healthy state.
The ready condition or path specified in the selector is not yet populated in the resource’s status.
The path value contains a typo, or dots in a key name are not escaped (data.tls\.crt).
The labels on the referenced resource do not match the selector labels.
The resolved value is present but is nil or an empty string (which is treated as unresolved).

Diagnosis:

# Check which selector is pending
kubectl get overlay <overlay-name> -o jsonpath='{.status.message}'

# Verify the referenced resource exists with the correct labels
kubectl get supervisorservice -l services.vcfa.broadcom.com/context=install

# Check if the ready condition is populated
kubectl get supervisorservice <service-name> -o jsonpath='{.status.readyConditions}' | jq

# Check the overlay controller logs
kubectl logs deployment/vcf-service-manager | grep <overlay-name>

Resolution:

Wait for the referenced resource to reach Healthy and populate the expected status field.
If the path is wrong, update spec.data.<key>.selector.path on the Overlay CR.
If labels do not match, correct the selector labels on the Overlay CR.

Target CR stuck in `Busy` - waiting for overlay (eager mode)

Symptom:

Status: Busy
Message: "Waiting for overlay: Overlay secret 'install.us-west-1' not found or not ready"

Likely causes:

The Overlay CR with the matching label does not exist.
The Overlay CR exists but is itself in Busy state (see above).
The services.vcfa.broadcom.com/overlay label value on the target CR does not exactly match the label on the Overlay CR.

Diagnosis:

# Check if the Overlay CR exists with the correct label
kubectl get overlay -l services.vcfa.broadcom.com/overlay=install.us-west-1

# Check Overlay status
kubectl get overlay <overlay-name> -o jsonpath='{.status.status}'

# Check if the Overlay Secret exists
kubectl get secret -l services.vcfa.broadcom.com/overlay=install.us-west-1

Resolution:

If the Overlay CR is missing, create it.
If it is stuck in Busy, resolve the underlying selector issue (see above).
If the configuration is optional and the service should proceed without it, change the overlay-apply label to lazy on the target CR.

ytt transformation fails

Symptom:

Status: Unhealthy
Message: "Failed to apply overlay: ytt transformation failed:
          overlay: Expected node at path 'values.arcturus' to exist"

Likely cause: The #@overlay/match pattern in the ytt template targets a path in the original authored spec (which contains selectors) rather than the runtime spec (which has selectors resolved to concrete values). For example, a template that matches {"values": {"selector": {}}} will fail because in the runtime spec values is already an expanded map, not a selector.

Diagnosis - reproduce locally:

# Extract runtime spec (base for ytt)
kubectl get secret <resource-name>-runtime-spec \
  -o jsonpath='{.data.runtimeSpec}' | base64 -d | yq -P > base.yml

# Extract overlay template
kubectl get secret <overlay-name> \
  -o jsonpath='{.data.overlay}' | base64 -d > overlay.yml

# Extract resolved values and add @data/values header
echo "#@data/values" > values.yml
echo "---" >> values.yml
kubectl get secret <overlay-name> \
  -o jsonpath='{.data.values}' | base64 -d | yq -P >> values.yml

# Run ytt locally to see the exact error
ytt -f base.yml -f overlay.yml -f values.yml

Resolution: Update the #@overlay/match pattern to target the resolved runtime spec structure.

# Incorrect - matches original spec structure (has selectors)
#@overlay/match by=overlay.subset({"values": {"selector": {}}})

# Correct - matches runtime spec structure (selectors already resolved)
#@overlay/match by=overlay.subset({"values": {"arcturus": {}}})

Overlay values not updating after source value changes

Symptom: Target CR is deployed with stale values even though the source resource’s value has changed.

Likely causes:

The Overlay is in Healthy state and will not re-evaluate selectors for up to 10 minutes.
The Overlay controller is not running.

Diagnosis:

# Check the current value in the source resource
kubectl get supervisorservice <service-name> \
  -o jsonpath='{.status.readyConditions.externalIp}'

# Check the value currently stored in the Overlay Secret
kubectl get secret <overlay-name> \
  -o jsonpath='{.data.values}' | base64 -d | jq '.depot_ip'

# Check when the Overlay last reconciled
kubectl get overlay <overlay-name> \
  -o jsonpath='{.status.conditions[?(@.type=="Ready")].lastTransitionTime}'

Resolution:

# Force an immediate Overlay reconciliation
kubectl annotate overlay <overlay-name> force-reconcile="$(date +%s)"

# If the target resource also needs to reconcile immediately
kubectl annotate supervisorservice <resource-name> force-reconcile="$(date +%s)"

Best Practices

Designing overlays

Write #@overlay/match patterns against the runtime spec structure, not the original authored spec. Verify by extracting the runtime spec and testing the ytt transformation locally before deploying.
Escape dots in JSONPath key names: data.tls\.crt not data.tls.crt.
Verify selector paths with kubectl get <resource> -o jsonpath='{.<path>}' before writing the Overlay CR.
Use expects="0+" in #@overlay/match for optional structural matches that may or may not be present.
Use descriptive data value names (depot_ip, tls_certificate) rather than generic ones (ip1, cert).

Choosing application mode

Use `eager` when…	Use `lazy` when…
The overlay contains values that the CR cannot function without	The overlay injects optional post-deployment configuration
Deploying without the overlay would create an inconsistent state	The CR can operate correctly with its base runtime spec
The overlay dependency must be resolved before any operations run	The overlay values can be applied on the next reconciliation loop

Overlay naming convention

Use a consistent naming scheme that encodes the context and scope:

<context>.<region>              →  install.us-west-1
<context>.<region>.<supervisor> →  publish.us-west-1.supervisor-vc1

This makes label matching unambiguous and simplifies debugging when multiple overlays exist within the same service.

Monitoring

Alert when an Overlay has been in Busy state for more than 5 minutes - this indicates a selector dependency is not resolving.
Monitor the time between a source value change and its propagation to the target CR to detect reconciliation delays.

Overlay CR reference - complete spec and status field documentation for the Overlay CR
SupervisorService - source of readyConditions values commonly used in overlay selectors
Extensions Overview - selector types (PathSelector, KindSelector) used in overlay data values

This site is open source. Improve this page.

vcf-services

Runtime Spec and Overlay System

Key Concepts

Runtime Spec

What It Contains

How Selectors Are Resolved

Overlay Application

Overlay System

How Overlays Work

Linking an Overlay to a Target CR

Application Modes

eager (default)

lazy

Overlay Data Values

Value Types

Resolution Behaviour

Reconciliation Intervals

ytt Overlay Templates

Basic structure

Matching strategies

Complete example - inject dynamic IP into SupervisorService

End-to-End Flow

Value change propagation latency

Troubleshooting

Overlay stuck in Busy - selector values not resolving

Target CR stuck in Busy - waiting for overlay (eager mode)

ytt transformation fails

Overlay values not updating after source value changes

Best Practices

Designing overlays

Choosing application mode

Overlay naming convention

Monitoring

Related Documents

`eager` (default)

`lazy`

Overlay stuck in `Busy` - selector values not resolving

Target CR stuck in `Busy` - waiting for overlay (eager mode)