Tasks

A task is a Python function that runs remotely in a container. Tasks are the unit of work in Flyte: they are versioned, cached, retried, and recorded, so a run you start today can be reproduced and inspected later.

Every task belongs to a TaskEnvironment, which declares the container image, the resources, and the secrets the task needs. You define the environment once and reuse it across the tasks that share it.

env = flyte.TaskEnvironment(name="etl", image=flyte.Image.from_debian_base())

@env.task
def extract(url: str) -> str:
    ...

Tasks compose. Calling one task from another builds the graph as your code executes, so fanout, branching, and error handling are ordinary Python rather than a separate DSL.

A task usually runs in a single container, but it doesn’t have to. For distributed workloads such as multi-node model training, a flyte.clustered.ClusteredTaskEnvironment runs one task across a gang of pods at once, wiring up a torchrun rendezvous so your task body executes on every worker:

import flyte
from flyte.clustered import ClusteredTaskEnvironment, TorchRun

env = ClusteredTaskEnvironment(
    name="ddp_env",
    image=flyte.Image.from_debian_base().with_pip_packages("torch"),
    replicas=2,            # number of pods (nodes)
    nproc_per_node=1,      # worker processes per pod
    runtime=TorchRun(),
)

@env.task
async def train() -> float:
    import torch.distributed as dist
    dist.init_process_group()  # RANK / WORLD_SIZE / MASTER_ADDR already set
    ...

See Clustered task environment for the full guide.

The three sections below follow the order you meet them in: describe the environment a task runs in, write the task logic, then get it onto a cluster.

Related

Tasks are also the substrate for the other two building blocks. An app serves a task’s results over HTTP; an agent drives tasks in a loop.