Why you need to evaluate llm applications

What different ways exist to evaluate LLM applications

The typical workflow for evaluating an LLM applications

How to create a test set for evaluating LLM applications

How to

Github

Book a Demo

Agenta AI

Guides

Reference

Changelog

Roadmap

Start for Free

Request a demo

How to automatically evaluate LLM applications

The open-source end-to-end LLMOps platform.

What is Agenta?

Create and deploy your first LLM app in one minute

Quick Start

Creating an LLM App

Learn how to use your custom application with Agenta

Creating Custom Applications

Create, experiment, and evaluate your applications all from the CLI

Using Agenta from CLI

Using the playground for prompt engineering.

Prompt Engineering

Learn how to integrate Agenta with your application

Deployment

Integrate applications and prompts created in agenta into your projects.

Integrating with agenta

Test Sets

Systematically evaluate your LLM applications and compare their performance.

Evaluating LLM Apps

Write the code for a custom evaluator on Agenta

Writing Custom Evaluators

Writing Webhook Evaluators

Human Evaluation

How to set up Agenta on your local computer

Deploy  Locally

Deploy on a Remote Instance

Deploy on AWS EC2

Installing Agenta on GCP Compute Engine using terraform

Deploy on Google Cloud Engine

Deploy on Kubernetes

Guide for migrating data from MongoDB (`agenta <v0.18`) to PostgreSQL (`agenta >=v0.19`).

Migration to PostgreSQL

This is a step-by-step guide for upgrading to the latest version of Agenta

Migration in MongoDB (Deprecated)

Team Management

Getting Support

Guidelines for contributing to the Agenta project

Contributing to Agenta

Some pointers to start contributing in Agenta

Development Mode in Agenta

How to File an Issue

You can use our SDK to build your own applications

Python SDK

Call at the start of your code to initialize your variant

init()

Use the Config Object to Access the Configuration Parameters of your Variant

The Config Object

Register the default configuration for your application

config.register_default()

config.pull()

config.push()

Data types allow you to configure the playground

Config Data Types

Agenta's CLI enables developers to easily create, evaluate, and deploy LLM apps from code. Install it here

Install

Deploy and test LLM apps using agenta CLI

Quick usage

agenta init

Serve an application to the agenta platform

agenta variant serve

agenta variant list

agenta variant remove

List all API keys associated with the authenticated user.

Args:
    request (Request): The incoming request object.

Returns:
    List[ListAPIKeysResponse]: A list of API Keys associated with the user.

List Api Keys

Creates an API key for a user.

Args:
    request (Request): The request object containing the user ID in the request state.

Returns:
    str: The created API key.

Create Api Key

Delete an API key with the given key prefix for the authenticated user.

Args:
    key_prefix (str): The prefix of the API key to be deleted.
    request (Request): The incoming request object.

Returns:
    dict: A dictionary containing a success message upon successful deletion.

Raises:
    HTTPException: If the API key is not found or does not belong to the user.

Delete Api Key

This Function is called by the CLI and is used to validate an API key provided by a user in agenta init setup.
Returns:
    bool: True. If the request reaches this point, the API key is valid.

Validate Api Key

Returns a list of organizations associated with the user's session.

Args:
    stoken_session (SessionContainer): The user's session token.

Returns:
    list[Organization]: A list of organizations associated with the user's session.

Raises:
    HTTPException: If there is an error retrieving the organizations from the database.

List Organizations

Create Organization

Get User Organization

Get an organization's details.

Raises:
    HTTPException: _description_
    Permission Denied

Returns:
    OrganizationDB Instance

Fetch Organization Details

Update Organization

Invite User To Workspace

Resend an invitation to a user to an Organization.

Raises:
    HTTPException: _description_; status_code: 500
    HTTPException: Invitation not found or has expired; status_code: 400
    HTTPException: You already belong to this organization; status_code: 400

Returns:
    JSONResponse: Resent invitation to user; status_code: 200

Resend Workspace Invitation

Accept an invitation to a workspace.

Raises:
    HTTPException: _description_; status_code: 500
    HTTPException: Invitation not found or has expired; status_code: 400
    HTTPException: You already belong to this organization; status_code: 400

Returns:
    JSONResponse: Accepted invitation to workspace; status_code: 200

Accept Workspace Invitation

Create Workspace

Update Workspace

Get all workspace roles.

Returns a list of all available workspace roles.

Returns:
    List[WorkspaceRoleResponse]: A list of WorkspaceRole objects representing the available workspace roles.

Raises:
    HTTPException: If an error occurs while retrieving the workspace roles.

Get All Workspace Roles

Get all workspace permissions.

Returns a list of all available workspace permissions.

Returns:
    List[Permission]: A list of Permission objects representing the available workspace permissions.

Raises:
    HTTPException: If there is an error retrieving the workspace permissions.

Get All Workspace Permissions

Assigns a role to a user in a workspace.

Args:
    payload (UserRole): The payload containing the organization id, user email, and role to assign.
    workspace_id (str): The ID of the workspace.
    request (Request): The FastAPI request object.

Returns:
    bool: True if the role was successfully assigned, False otherwise.

Raises:
    HTTPException: If the user does not have permission to perform this action.
    HTTPException: If there is an error assigning the role to the user.

Assign Role To User

Delete a role assignment from a user in a workspace.

Args:
    workspace_id (str): The ID of the workspace.
    email (str): The email of the user to remove the role from.
    org_id (str): The ID of the organization.
    role (str): The role to remove from the user.
    request (Request): The FastAPI request object.

Returns:
    bool: True if the role assignment was successfully deleted.

Raises:
    HTTPException: If there is an error in the request or the user does not have permission to perform the action.
    HTTPException: If there is an error in updating the user's roles.

Unassign Role From User

Remove a user from a workspace.

Parameters:
- payload (UserRole): The payload containing the user email and organization ID.
- workspace_id (str): The ID of the workspace.
- request (Request): The FastAPI request object.

Returns:
- WorkspaceResponse: The updated workspace.

Raises:
- HTTPException: If the user does not have permission to perform this action.
- HTTPException: If there is an error during the removal process.

Getting Started

Prompt Management and Engineering

Evaluation

Observability

Self-host agenta

Misc

How to automatically evaluate LLM applications

Why you need to evaluate llm applications

What different ways exist to evaluate LLM applications

The typical workflow for evaluating an LLM applications

How to create a test set for evaluating LLM applications

How to

Getting Started

Prompt Management and Engineering

Evaluation

Observability

Self-host agenta

Misc

​Why you need to evaluate llm applications

​What different ways exist to evaluate LLM applications

​The typical workflow for evaluating an LLM applications

​How to create a test set for evaluating LLM applications

​How to

Why you need to evaluate llm applications

What different ways exist to evaluate LLM applications

The typical workflow for evaluating an LLM applications

How to create a test set for evaluating LLM applications

How to