Deploy the dataplane

If you have not yet set up the required resources (Kubernetes cluster, object storage, container registry, credentials), see Prepare infrastructure first.

Assumptions

• You have a Union.ai organization, and you know the control plane URL for your organization (e.g. https://your-org-name.us-east-2.unionai.cloud).
• You have a cluster name provided by or coordinated with Union.
• You have a Kubernetes cluster, running one of the most recent three minor Kubernetes versions. Learn more.
• Object storage provided by a vendor or an S3-compatible platform (such as MinIO), with CORS configured as described in Prepare infrastructure.
• A container registry accessible from your cluster.

Prerequisites

• Install Helm 3.
• Install uctl.
• Install the flyte CLI.
• Install the flyteplugins-union plugin, which provides the flyte create cluster and flyte get cluster commands: pip install flyteplugins-union.

Deploy the Union.ai operator

1. Add the Union.ai Helm repo:

   helm repo add unionai https://unionai.github.io/helm-charts/
   helm repo update

2. Configure the flyte CLI to talk to your control plane, then register the cluster name:

   flyte create config --endpoint <YOUR_UNION_CONTROL_PLANE_URL> --org <YOUR_ORG_NAME>
   flyte create cluster <YOUR_SELECTED_CLUSTERNAME> --pool default

   flyte create config writes .flyte/config.yaml. The first command that contacts the control plane opens a browser to authenticate you.

   The second command registers the cluster before you install the chart: the data plane binds to this record when it starts. Every organization is provisioned with a default pool, so --pool default needs no extra setup.

3. Use the uctl selfserve provision-dataplane-resources command to generate a new client and client secret for communicating with your Union control plane, provision authorization permissions for the app to operate on the Union cluster name you have selected, and provide follow-up instructions:

   uctl config init --host=<YOUR_UNION_CONTROL_PLANE_URL>
   uctl selfserve provision-dataplane-resources --clusterName <YOUR_SELECTED_CLUSTERNAME>  --provider custom

   • The command will output the ID, name, and a secret that will be used by the Union services to communicate with your control plane. You will pass the client ID and client secret to the Helm chart in step 5.

   • Save the secret that is displayed. Union does not store it, and it cannot be retrieved later.

4. Download the base values file for the data plane chart and fill in your infrastructure details:

   curl -O https://raw.githubusercontent.com/unionai/helm-charts/main/charts/dataplane/values.yaml

   The published overlays cover AWS, GCP, and Azure only, so a generic or on-premise environment starts from the base values file:

   • Set global.UNION_CONTROL_PLANE_HOST and global.CONTROLPLANE_HOST to your control plane hostname (no scheme, no port).
   • Set global.CLUSTER_NAME to the cluster name you registered in step 2.
   • Set global.ORG_NAME to your organization name.
   • Set storage.endpoint to your S3-compatible storage endpoint (e.g. your MinIO URL).
   • Set storage.bucketName and storage.fastRegistrationBucketName to your bucket name(s).
   • Set storage.region to the region of your storage provider.
   • The same credentials are also needed in fluentbit.env for log shipping.

   Rather than putting your storage credentials in the values file, store them in a Kubernetes Secret and reference it from the chart. Create the namespace and the secret first; the chart reads the secret while rendering, so it must exist before you install:

   kubectl create namespace union
   kubectl create secret generic storage-credentials -n union \
     --from-literal=access_key_id=<ACCESS_KEY_ID> \
     --from-literal=secret_key=<SECRET_ACCESS_KEY>

   Then reference it instead of setting storage.accessKey and storage.secretKey:

   storage:
     provider: compat
     authType: accesskey
     credentialsSecretRef:
       name: storage-credentials

   The chart resolves the secret with a Helm lookup, which returns nothing during helm template or --dry-run. Those commands render the storage config without credentials; only a real install picks them up. If your secret uses different field names, set credentialsSecretRef.accessKeyIdKey and credentialsSecretRef.secretKeyKey to match.

5. Install the data plane Helm chart, passing the client ID and client secret from step 3:

   helm upgrade --install union unionai/dataplane \
     -f values.yaml \
     --set global.AUTH_CLIENT_ID=<CLIENT_ID> \
     --set-string secrets.admin.clientId=<CLIENT_ID> \
     --set secrets.admin.clientSecret=<CLIENT_SECRET> \
     --namespace union \
     --create-namespace 

6. Once deployed you can check to see if the cluster has been successfully registered to the control plane:

   flyte get cluster

   The command groups clusters by state. A successfully registered cluster appears under Enabled Clusters:

   Enabled Clusters
   NAME        ORG     STATE     HEALTH
   <cluster>   <org>   enabled   healthy

7. Follow the Quickstart to run your first workflow and verify your cluster is working correctly.

Next: manage your cluster and pools

uctl selfserve provision-dataplane-resources provisions the data plane and registers this cluster with the control plane. Once it is connected, you manage the cluster pool it belongs to, and route work to it with queues, from the Cluster and workload management user guide:

• Cluster pools: group clusters that share one data plane (object store, secrets, registry).
• Clusters: inspect and manage the cluster records registered with the control plane.
• Managing queues: route workloads to a pool and enforce concurrency, priority, and fairness.

Each cluster is assigned exactly one pool. If no custom pool is specified when the cluster is created, it joins the default pool that every organization is provisioned with, so a single-cluster deployment needs no extra pool setup.