Skip to main content

LiteLLM

LiteLLM is Python SDK that allows you to call 100+ LLM APIs in OpenAI format - [Bedrock, Azure, OpenAI, VertexAI, Cohere, Anthropic, Sagemaker, HuggingFace, Replicate, Groq].

This guide shows you how to instrument LiteLLM applications using Agenta's observability features.

Installation

Install the required packages:

pip install -U agenta litellm

Configure Environment Variables

import os

os.environ["AGENTA_API_KEY"] = "YOUR_AGENTA_API_KEY"
os.environ["AGENTA_HOST"] = "https://cloud.agenta.ai"

Code Example

import agenta as ag
import litellm
import asyncio

ag.init()

@ag.instrument()
async def agenerate_completion():
litellm.callbacks = [ag.callbacks.litellm_handler()]

# Define the messages for the chat completion
messages = [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Write a short story about AI Engineering."},
]
temperature = 0.2
max_tokens = 100

# Make an asynchronous chat completion request
chat_completion = await litellm.acompletion(
model="gpt-3.5-turbo",
messages=messages,
temperature=temperature,
max_tokens=max_tokens,
)
print(chat_completion)

# Run the asynchronous function
asyncio.run(agenerate_completion())

Explanation

  • Initialize Agenta: ag.init() sets up the Agenta SDK.
  • Instrument the Function: The @ag.instrument() decorator wraps the agenerate_completion function, creating a parent span in Agenta.
  • Set Up Callback: litellm.callbacks = [ag.callbacks.litellm_handler()] sets the Agenta callback handler for LiteLLM. This enables LiteLLM to send trace data to Agenta.