Distributed Tracing with OpenTelemetry

Agenta provides built-in OpenTelemetry instrumentation to simplify distributed tracing in your applications. This guide explains how to implement and manage distributed tracing using the Agenta SDK, and how to integrate external tracing setups with Agenta.

Using OpenTelemetry with Agenta

Agenta supports distributed tracing out of the box when using the provided SDK functions:

1. Sending Requests (Propagation)

When making requests to other services or sub-systems, use agenta.tracing.inject() to inject necessary headers:

method = "POST"
url = "https://example-service/api"
params = {}
headers = agenta.tracing.inject()  # automatically injects 'Authorization', 'Traceparent', 'Baggage'
body = {"key": "value"}

response = requests.request(
    method=method,
    url=url,
    params=params,
    headers=headers,
    json=body,
)

The agenta.tracing.inject() function returns headers containing:

• Authorization: Authentication information
• Traceparent: Identifies the current trace and span
• Baggage: Contains application-specific context

These headers can be modified before sending them as part of the request if needed.

2. Receiving Requests (Extraction)

Agenta simplifies receiving and handling incoming trace contexts:

• If you're using ag.route() and ag.instrument(), extraction is automatic.
• For manual extraction, use agenta.tracing.extract():

traceparent, baggage = agenta.tracing.extract()  # includes 'Traceparent', 'Baggage'

# Use traceparent and baggage to set up your OpenTelemetry context
# (Implementation depends on your specific use case)

note

extract() does not provide Authorization because there are many authentication methods (apikey, bearer, secret, access tokens), each requiring different handling. The middlewares and decorators in the Agenta SDK handle this automatically when you use ag.route() and ag.instrument().

OpenTelemetry Tracing without Agenta SDK

If you're working with systems that don't use the Agenta SDK, you can still integrate with Agenta's tracing infrastructure using standard OpenTelemetry.

1. Setup Requirements

Install dependencies:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp

2. Configure Environment Variables

# OTEL_PROPAGATORS = unset or "tracecontext,baggage"
# OTEL_EXPORTER_OTLP_COMPRESSION = unset or "gzip"
# OTEL_EXPORTER_OTLP_ENDPOINT = "https://cloud.agenta.ai/api/otlp"
# OTEL_EXPORTER_OTLP_HEADERS = "authorization=ApiKey xxx"
# OTEL_EXPORTER_OTLP_TRACES_ENDPOINT = "https://cloud.agenta.ai/api/otlp/v1/traces"
# OTEL_EXPORTER_OTLP_TRACES_HEADERS = "authorization=ApiKey xxx"

3. Setup in Code

from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
from opentelemetry.baggage.propagation import W3CBaggagePropagator
from opentelemetry.sdk.trace import TracerProvider, Span
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter, Compression

# Configuration
endpoint = "https://cloud.agenta.ai/api/otlp/v1/traces"
compression = Compression.Gzip
headers = {
    "traceparent": "00-xxx-xxx-01",
    "baggage": "ag.refs.application.id=xxx",
    "authorization": "ApiKey xxx",
}

# Set up provider, processor, and tracer
provider = TracerProvider()

processor = BatchSpanProcessor(
    OTLPSpanExporter(
        endpoint=endpoint,
        headers={"authorization": headers["authorization"]},
        compression=compression,
    )
)

provider.add_span_processor(processor)

tracer = provider.get_tracer("agenta.tracer")

# Extract incoming trace context
carrier = {"traceparent": headers["traceparent"]}
context = TraceContextTextMapPropagator().extract(carrier=carrier, context=None)

carrier = {"baggage": headers["baggage"]}
context = W3CBaggagePropagator().extract(carrier=carrier, context=context)

# Create and use spans
with tracer.start_as_current_span(name="agenta", context=context) as span:
    span: Span

    print(hex(span.get_span_context().trace_id))
    print(hex(span.get_span_context().span_id))
    print(span.name)

Using an OTEL Collector

If you're using an OpenTelemetry collector, you can configure it to export traces to Agenta. Here's a sample configuration (otel-collector-config.yml):

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch: {}

exporters:
  otlphttp/agenta:
    endpoint: "https://cloud.agenta.ai/api/otlp"

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlphttp/agenta]

With this configuration:

• The collector receives traces via OTLP/HTTP on port 4318
• It batches the spans for efficiency
• It exports them to Agenta's OTLP endpoint

Span Attributes

When using OpenTelemetry without the Agenta SDK, you need to manually set the appropriate attributes on your spans to integrate properly with Agenta's ecosystem.

Namespace Convention

Agenta uses the ag.* namespace for its attributes. Here are the key namespaces:

• ag.refs.*: References to Agenta entities (applications, etc.)
• ag.data.*: Input, internal, and output data
• ag.metrics.*: Performance metrics and costs

Examples

# Reference to Agenta application
span.set_attribute("ag.refs.application.id", AGENTA_APPLICATION_ID)

# Data attributes
span.set_attribute("ag.data.inputs.key", "Hello,")
span.set_attribute("ag.data.internals.key", "(Leo)")
span.set_attribute("ag.data.outputs.key", "World!")

# Metrics - unit values
span.set_attribute("ag.metrics.unit.some_key", 3)
span.set_attribute("ag.metrics.acc.some_key", 15)

# Cost and token metrics
span.set_attribute("ag.metrics.unit.costs.total", 1)
span.set_attribute("ag.metrics.unit.tokens.total", 100)

info

Apart from these custom attributes, standard OpenTelemetry events, links, status, and exceptions work as usual.