Roadmap

You can view prompt and messages in markdown both in the playground and in the observability drawer.

You can now upload images to the playground and use them in your prompts.

You can trace your calls from LlamaIndex in one line.

You can now use the annotation API to add annotations (e.g. scores, feedback) to LLM responses traced in Agenta.

You can now define and test tools in the playground. You can save tool definitions as part of your prompts.

We support now structured output in the playground. You can now define and validate structured output formats and save them as part of your prompt.

We are reworking the evaluation results page to make it more user friendly and informative.

Adding the ability to configure evaluators (llm-as-a-judge or custom) and run them automatically on new traces.

Until now evaluations were only available as managed by Agenta. We are now adding the ability to run evaluations programmatically through the SDK.

We are improving the speed of the playground especially with hundreds of revisions.

Fixing workspace bugs, especially with page refresh, workspace selection, and sharing workspace links.

We are adding the ability to filter traces by annotation. This is useful for finding traces with low scores or feedback.

We are making it easy to use and navigate in the playground with large test sets .

Using test cases from different test sets is not possible right now in the Playground. We are adding the ability to append a single test case to a test set.

We are reworking the test set view to make it easier to visualize and edit test sets.

We are adding the ability to cache prompts in the SDK.

We are adding the ability to version test sets. This is useful for correctly comparing evaluation results.

We are adding the ability to tag traces, test sets, evaluations and prompts. This is useful for organizing and filtering your data.

We are adding the ability to use built-in LLM tools (e.g. web search) in the playground.