Cluster and workload management

As a Union.ai deployment grows past a single cluster, you need to control where a workload runs and under what limits. Three primitives do this:

• Cluster pool: an isolation boundary. The clusters and queues inside a pool share one data plane configuration: the same object store, secret store, and container registry. Work cannot cross from one pool to another (see Crossing a pool boundary).
• Cluster: an execution cluster that lives in exactly one pool.
• Queue: what you submit work to. A queue lives in one pool, routes work to one or more clusters in that pool, and applies the concurrency, depth, priority, and fairness limits for the work it admits. Every cluster automatically gets a co-named queue that routes only to it, so any cluster can be targeted by name without creating anything (see Queues you get for free).

Tooling

Pools, clusters, and queues are managed with the flyte CLI or the flyteplugins.union.remote Python objects, and are set up by your platform administrator. These are administrative tasks; most workflow authors only need task-side queue routing.

Standing up a self-managed cluster?

The pages here manage the control-plane records for pools, clusters, and queues. They do not provision the data plane itself: the cloud resources (object store, secret store, registry) and the Helm release that registers a cluster with the control plane.

If you run a self-managed deployment, provision the data plane first with Self-managed deployment (for example, Data plane setup on AWS), then use the commands here to manage the pool, cluster, and queue records that route work to it.

How they fit together

            flowchart TD
    Org["Organization"]
    Org --> PD
    Org --> PP

    subgraph PD["Cluster pool: default"]
        direction TB
        QD["Queue: default<br/>selector: *"]
        CA["Cluster A"]
        CB["Cluster B"]
        QD --> CA
        QD --> CB
    end

    subgraph PP["Cluster pool: prod"]
        direction TB
        QP["Queue: prod-queue<br/>selector: [Cluster C, Cluster D]"]
        QG["Queue: gpu-queue<br/>selector: [Cluster C]"]
        CC["Cluster C"]
        CD["Cluster D"]
        QP --> CC
        QP --> CD
        QG --> CC
    end
        

A cluster pool is an isolation boundary: both clusters and queues live inside a pool, and everything in it shares one data plane. A queue routes work to one or more clusters in its own pool, and the three queues above show the routing choices you have:

• default uses the wildcard selector *: it spreads across every cluster in its pool that is healthy and enabled, and picks up new clusters automatically as they join. An unhealthy cluster stops receiving new work until it recovers — see Wildcard routing.
• prod-queue names both clusters in its pool explicitly. The result looks like the wildcard today, but the membership is frozen: a Cluster E added to prod later gets no work from this queue until you add it to the selector.
• gpu-queue names a single cluster, pinning that lane to Cluster C while Cluster D stays free for other work.

The key invariant: a queue can never reach a cluster outside its pool, because a run’s inputs, code, and secrets are uploaded to that pool’s data plane and no other pool’s clusters can read them. That is what makes a pool an isolation boundary.

Crossing a pool boundary

Because pools don’t share a data plane, they don’t connect. A run can never move from one pool to another. A queue can be reassigned to another pool only after it has been fully drained, so the move never carries in-flight work across the boundary. A cluster can be reassigned to another pool, but today only as a disruptive maintenance operation that stops nothing and reschedules nothing — a cluster-level drain that guarantees an idle cluster before the move is coming soon. See Move a cluster to a different pool. For work in flight, crossing a pool boundary means physically re-landing the workload in the destination pool’s data plane: moving its data, containers (images), code, and secrets into the new pool’s object store, registry, and secret store. This is deliberate friction: it keeps in-flight work from ever pointing at storage it can’t read, and it’s why pool changes are rare and explicit (moving work between pools is a drain-and-replace migration).

In this section