Advanced configurations

This section covers the configuration of Union.ai features on your data plane cluster.

Data planes connected from the Union.ai UI

Where a page sets values on the dataplane Helm chart, it is for a data plane installed with Helm: on OCI, CoreWeave, Crusoe, Nebius, or generic Kubernetes, or an existing AWS, GCP, or Azure deployment set up that way. On a data plane connected from the Union.ai UI, the agent installs the chart and manages its values, so change those settings through Union.ai rather than with helm upgrade. Pages that change your cloud resources or cluster, such as data retention, code viewer, and node pools, apply either way.

Interruptible task schedulingLet interruptible tasks run on spot instances, and handle reclamation.Node poolsSeparate Union services from worker pods so they do not contend for resources.Code ViewerLet the console show the exact code a task ran, transferred from the code bundle.Image BuilderBuild container images inside the data plane so tasks can use the remote builder.Multiple ClustersAttach several Kubernetes clusters to one control plane with the clusterPool abstraction.Persistent logsHow FluentBit collects container logs from every node and ships them for retention.MonitoringThe Prometheus instance the data plane deploys, and the metrics it collects.SecretsHow Union Secrets are stored as Kubernetes secrets, and what to configure.Data retention policiesImplications of object storage retention or lifecycle policies on the data plane bucket.Compute pluginsEnable the compute plugins that need cluster-side configuration, such as Dask, Ray, and Spark.Namespace mappingChange how project and domain pairs map onto Kubernetes namespaces.VolumesConfigure the durable, versioned file systems tasks mount and read and write.TelemetryTurn off the telemetry the Flyte SDK sends from your data plane with DO_NOT_TRACK.