OpenTelemetry

flyteplugins-otel turns a Flyte run into an OpenTelemetry trace.

Every task becomes a span. Every traced function becomes a child span inside it. Spans created by your own code or by any OpenTelemetry instrumentation library nest underneath without extra wiring. Export goes wherever OTLP goes: Grafana Tempo, Jaeger, Honeycomb, an OpenTelemetry Collector or several at once.

None of this is specific to agents or to LLM workloads. It is ordinary distributed tracing for ordinary Flyte tasks, plus two behaviors that exist because Flyte runs are durable and a stock OpenTelemetry setup has no way to model them:

• A crashed and resumed run is one trace, not several: Each attempt is a fresh process with a fresh OpenTelemetry SDK, so each would normally mint its own trace ID. The plugin derives the trace ID from the run instead, so every process converges on the same trace with no coordination.
• Steps served from the durable log still appear. A resumed run replays completed steps rather than re-executing them, so nothing instruments them and the trace would otherwise have holes exactly where durability did its job. The plugin records them as spans marked flyte.replayed.

Traces across crashes and resumes covers both in detail.

Installation

pip install flyteplugins-otel

OTLP over HTTP is included. gRPC ships separately:

pip install "flyteplugins-otel[grpc]"

Quick start

Call init() once at module scope, then write tasks as you normally would:

import flyte
from flyteplugins.otel import init

# Module scope, not inside a task. The task span opens before the task body runs,
# so initializing from within the body means that task's own span is already missed.
init(service_name="my-service")

env = flyte.TaskEnvironment(
    name="my_env",
    image=flyte.Image.from_debian_base().with_pip_packages("flyteplugins-otel"),
)


@flyte.trace
async def double(x: int) -> int:
    return x * 2


@env.task
async def main(n: int = 3) -> int:
    total = 0
    for i in range(n):
        total += await double(i)
    return total


if __name__ == "__main__":
    flyte.init_from_config()
    print(flyte.run(main, n=3).url)

That produces one trace shaped like this:

main                          ← task span
├── double                    ← flyte.trace step span
├── double
└── double

The quick start run in the Flyte UI. On the left, the action tree shows main and its three double steps. On the right, the Logs tab holds ConsoleSpanExporter output for one double span, carrying the flyte.* attributes and the parent_id that nests it under the task span.

With no arguments, init() reads the standard OTEL_EXPORTER_OTLP_ENDPOINT and OTEL_EXPORTER_OTLP_HEADERS variables, which is how most vendors document their setup. See Exporters and configuration for pointing it at a real backend and for supplying credentials as a flyte.Secret instead of hardcoding them.

What becomes a span

Task lifecycle itself is not instrumented: there are no spans for scheduling, queueing or the control plane’s decision to retry. A span starts when a container begins executing a task.

You will however, see HTTP client spans for Flyte’s own calls to the control plane once tracing is on. Those come from Flyte’s transport rather than from this plugin; Flyte’s own control-plane spans explains where they come from and how to switch them off.

Span attributes

Every span the plugin emits carries the identifiers needed to get back to the run that produced it. These names are effectively public API: a Grafana data link queries on them to jump from a span into the Flyte UI and GrafanaTrace is built on them.

Your own spans

Spans you create with a plain OpenTelemetry tracer nest inside the task span automatically. Parenting in OpenTelemetry comes from the active context and the plugin keeps the task span active for the whole task body, so there is nothing to extract and no context to pass around:

from opentelemetry import trace

tracer = trace.get_tracer("my.app")


@env.task
async def etl(rows: int = 100) -> int:
    with tracer.start_as_current_span("extract") as span:
        span.set_attribute("rows.requested", rows)
        extracted = rows

    with tracer.start_as_current_span("transform"):
        # Spans nest as deeply as you like; this one lands under transform.
        with tracer.start_as_current_span("validate"):
            transformed = extracted - 1

    with tracer.start_as_current_span("load") as span:
        span.set_attribute("rows.loaded", transformed)

    return transformed

The same is true of third-party auto-instrumentation. An HTTP client instrumentor, a database instrumentor or an LLM instrumentor needs no extra wiring:

from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor

init(service_name="my-service")
HTTPXClientInstrumentor().instrument()

Call init() before the other library so everything stays on one export pipeline. For libraries that want to be handed a tracer, get_tracer() returns the one the plugin built.

What’s next

• Exporters and configuration: point the plugin at a backend, adopt a tracer provider you already have, and link back from Grafana into the Flyte UI.
• Traces across crashes and resumes: trace context in and out of a run, run-derived trace IDs, and replayed steps.
• Grafana Agent Observability: add LLM generations, tool calls, token usage and cost on top of these traces.