Roadmap

Completely redesigned evaluation results dashboard with performance plots, side-by-side comparison, improved test cases view, focused detail view, configuration visibility, and run naming.

URLs now include workspace context, making them shareable between team members. Fixed workspace bugs with page refresh and workspace selection.

We improved the speed of the playground (creation of prompts, navigation, etc.) especially with hundreds of revisions.

You can view prompt and messages in markdown both in the playground and in the observability drawer.

You can now upload images to the playground and use them in your prompts.

You can trace your calls from LlamaIndex in one line.

You can now use the annotation API to add annotations (e.g. scores, feedback) to LLM responses traced in Agenta.

You can now define and test tools in the playground. You can save tool definitions as part of your prompts.

We support now structured output in the playground. You can now define and validate structured output formats and save them as part of your prompt.

Adding the ability to configure evaluators (llm-as-a-judge or custom) and run them automatically on new traces.

Until now evaluations were only available as managed by Agenta. We are now adding the ability to run evaluations programmatically through the SDK.

We are adding the ability to filter traces by annotation. This is useful for finding traces with low scores or feedback.

We are adding the ability to filter traces by date range in the metrics dashboard.

We are making it easy to use and navigate in the playground with large test sets .

Using test cases from different test sets is not possible right now in the Playground. We are adding the ability to append a single test case to a test set.

We are reworking the test set view to make it easier to visualize and edit test sets.

We are adding the ability to cache prompts in the SDK.

We are adding the ability to version test sets. This is useful for correctly comparing evaluation results.

We are adding the ability to tag traces, test sets, evaluations and prompts. This is useful for organizing and filtering your data.

We are adding the ability to use built-in LLM tools (e.g. web search) in the playground.