URLs across Agenta now include workspace context, making them fully shareable between team members. This was a highly requested feature that addresses several critical issues with the previous URL structure.
URLs across Agenta now include workspace context, making them fully shareable between team members. Previously, URLs would always point to the default workspace, causing issues when refreshing pages or sharing links.
Now you can deep link to almost anything in the platform - prompts, evaluations, and more - in any workspace. Share links directly with team members and they'll see exactly what you intended, regardless of their default workspace settings.
We rewrote most of Agenta's frontend. You'll see much faster speeds when you create prompts or use the playground.
We also made many improvements and fixed bugs:
Improvements:
LLM-as-a-judge now uses double curly braces {{}} instead of single curly braces { and }. This matches how normal prompts work. Old LLM-as-a-judge prompts with single curly braces still work. We updated the LLM-as-a-judge playground to make editing prompts easier.
We rebuilt the human evaluation workflow from scratch. Now you can set multiple evaluators and metrics and use them to score the outputs.
This lets you evaluate the same output on different metrics like relevance or completeness. You can also create binary, numerical scores, or even use strings for comments or expected answer.
Watch the video below and read the post for more details. Or check out the docs to learn how to use the new human evaluation workflow.
We've made our product roadmap completely transparent and community-driven.
You can now see exactly what we're building, what's shipped, and what's coming next. Plus vote on features that matter most to you.
Why we're doing this: We believe open-source startups succeed when they create the most value possible, and the best way to do that is by building with our community, not in isolation. Up until now, we've been secretive with our roadmap, but we're losing something important: your feedback and the ability to let you shape our direction. Today we're open-sourcing our roadmap because we want to build a community of owners, not just passive users.
We've made significant improvements across Agenta with a major documentation overhaul, new model support, self-hosting enhancements, and UI improvements.
Revamped Prompt Engineering Documentation:
We've completely rewritten our prompt management and prompt engineering documentation.
Start exploring the new documentation in our updated Quick Start Guide.
New Model Support:
Our platform now supports several new LLM models:
Google's Gemini 2.5 Pro and Flash
Alibaba Cloud's Qwen 3
OpenAI's GPT-4.1
These models are available in both the playground and through the API.
Playground Enhancements:
We've added a draft state to the playground, providing a better editing experience. Changes are now clearly marked as drafts until committed.
Self-Hosting Improvements:
We've significantly simplified the self-hosting experience by changing how environment variables are handled in the frontend:
No more rebuilding images to change ports or domains
Dynamic configuration through environment variables at runtime
We are SOC 2 Type 2 Certified. This means that our platform is audited and certified by an independent third party to meet the highest standards of security and compliance.
We've introduced the Prompt and Deployment Registry, giving you a centralized place to manage all variants and versions of your prompts and deployments.
Key capabilities:
View all variants and revisions in a single table
Access all commits made to a variant
Use older versions of variants directly in the playground
We've made several improvements to the playground, including:
Improved scrolling behavior
Increased discoverability of variants creation and comparison
Implemented stop functionality in the playground
As for custom workflows, now they work with sub-routes. This means you can have multiple routes in one file and create multiple custom workflows from the same file.
We've introduced major improvements to Agenta, focusing on OpenTelemetry compliance and simplified custom workflow debugging.
OpenTelemetry (OTel) Support:
Agenta is now fully OpenTelemetry-compliant. This means you can seamlessly integrate Agenta with thousands of OTel-compatible services using existing SDKs. To integrate your application with Agenta, simply configure an OTel exporter pointing to your Agenta endpoint—no additional setup required.
We've enhanced distributed tracing capabilities to better debug complex distributed agent systems. All HTTP interactions between agents—whether running within Agenta's SDK or externally—are automatically traced, making troubleshooting and monitoring easier.
Based on your feedback, we've streamlined debugging and running custom workflows:
Run workflows from your environments: You no longer need the Agenta CLI to manage custom workflows. Setting up custom workflows now involves simply adding the Agenta SDK to your code, creating an endpoint, and connecting it to Agenta via the web UI. You can check how it's done in the quick start guide.
Custom Workflows in the new playground: Custom workflows are now fully compatible with the new playground. You can now nest configurations, run side-by-side comparisons, and debug your agents and complex workflows very easily.
We've rebuilt our playground from scratch to make prompt engineering faster and more intuitive. The old playground took 20 seconds to create a prompt - now it's instant.
Key improvements:
Create prompts with multiple messages using our new template system
Format variables easily with curly bracket syntax and a built-in validator
Switch between chat and completion prompts in one interface
Load test sets directly in the playground to iterate faster
Save successful outputs as test cases with one click
Compare different prompts side-by-side
Deploy changes straight to production
For developers, now you create prompts programmatically through our API.
You can explore these features in our updated playground documentation.
We've achieved SOC 2 Type 1 certification, validating our security controls for protecting sensitive LLM development data. This certification covers our entire platform, including prompt management, evaluation frameworks, and observability tools.
Key security features and improvements:
Data encryption in transit and at rest
Enhanced access control and authentication
Comprehensive security monitoring
Regular third-party security assessments
Backup and disaster recovery protocols
This certification represents a significant milestone for teams using Agenta in production environments. Whether you're using our open-source platform or cloud offering, you can now build LLM applications with enterprise-grade security confidence.
We've also updated our trust center with detailed information about our security practices and compliance standards. For teams interested in learning more about our security controls or requesting our SOC 2 report, please contact team@agenta.ai.
This release introduces the ability to add spans to test sets, making it easier to bootstrap your evaluation data from production. The new feature lets you:
Add individual or batch spans to test sets
Create custom mappings between spans and test sets
Preview test set changes before committing them
Additional improvements:
Fixed CSV test set upload issues
Prevented viewing of incomplete evaluations
Added mobile compatibility warning
Added support for custom ports in self-hosted installations
You can now see traces directly in the playground. For simple applications, this means you can view the prompts sent to LLMs. For custom workflows, you get an overview of intermediate steps and outputs. This makes it easier to understand what’s happening under the hood and debug your applications.
We’ve strengthened authentication for deployed applications. As you know, Agenta lets you either fetch the app’s config or call it with Agenta acting as a proxy. Now, we’ve added authentication to the second method. The APIs we create are now protected and can be called using an API key. You can find code snippets for calling the application in the overview page.
We’ll publish a full blog post soon, but here’s a quick look at what the new observability offers:
A redesigned UI that lets you visualize nested traces, making it easier to understand what’s happening behind the scenes.
The web UI lets you filter traces by name, cost, and other attributes—you can even search through them easily.
The SDK is Otel-compatible, and we’ve already tested integrations for OpenAI, LangChain, LiteLLM, and Instructor, with guides available for each. In most cases, adding a few lines of code will have you seeing traces directly in Agenta.
Next: Prompt Management
We’ve completely rewritten the prompt management SDK, giving you full CRUD capabilities for prompts and configurations. This includes creating, updating, reading history, deploying new versions, and deleting old ones. You can find a first tutorial for this here.
And finally: LLM-as-a-Judge Overhaul
We’ve made significant upgrades to the LLM-as-a-Judge evaluator. It now supports prompts with multiple messages and has access to all variables in a test case. You can also switch models (currently supporting OpenAI and Anthropic). These changes make the evaluator much more flexible, and we’re seeing better results with it.
We updated the Application Management View to improve the UI. Many users struggled to find their applications when they had a large number, so we've improved the view and added a search bar for quick filtering.
Additionally, we are moving towards a new project structure for the application. We moved test sets and evaluators outside of the application scope. So now, you can use the same test set and evaluators in multiple applications.
Bug Fixes
Added an export button in the evaluation view to export results from the main view.
Eliminated Pydantic warnings in the CLI.
Improved error messages when fetch_config is called with wrong arguments.
Enhanced the custom code evaluation sandbox and removed the limitation that results need to be between 0 and 1
Many users faced challenges configuring evaluators in the web UI. Some
evaluators, such as LLM as a judge, custom code, or RAG evaluators can be
tricky to set up correctly on the first try. Until now, users needed to setup,
run an evaluation, check the errors, then do it again.
To address this, we've introduced a new evaluator test/debug playground. This feature allows you to test the evaluator live on real data, helping you test the configuration before committing to it and using it for evaluations.
Additionally, we have improved and redesigned the evaluation view. Both automatic and human evaluations are now within the same view but in different tabs. We're moving towards unifying all evaluator results and consolidating them in one view, allowing you to quickly get an overview of what's working.
We've completely redesigned the platform's UI. Additionally we have introduced a new overview view for your applications. This is part of a series of upcoming improvements slated for the next few weeks.
The new overview view offers:
A dashboard displaying key metrics of your application
A table with all the variants of your applications
A summary of your application's most recent evaluations
We've also added a new JSON Diff evaluator. This evaluator compares two JSON objects and provides a similarity score.
Lastly, we've updated the UI of our documentation.
We've released a new version of the SDK for creating custom applications. This Pydantic-based SDK significantly simplifies the process of building custom applications. It's fully backward compatible, so your existing code will continue to work seamlessly. We'll soon be rolling out comprehensive documentation and examples for the new SDK.
In the meantime, here's a quick example of how to use it:
import agenta as ag from agenta import Agenta from pydantic import BaseModel, Field ag.init() # Define the configuration of the application (that will be shown in the playground ) classMyConfig(BaseModel): temperature:float= Field(default=0.2) prompt_template:str= Field(default="What is the capital of {country}?") # Creates an endpoint for the entrypoint of the application @ag.route("/", config_schema=MyConfig) defgenerate(country:str)->str: # Fetch the config from the request config: MyConfig = ag.ConfigManager.get_from_route(schema=MyConfig) prompt = config.prompt_template.format(country=country) chat_completion = client.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role":"user","content": prompt}], temperature=config.temperature, ) return chat_completion.choices[0].message.content
We're excited to announce two major features this week:
We've integrated RAGAS evaluators into agenta. Two new evaluators have been added: RAG Faithfulness (measuring how consistent the LLM output is with the context) and Context Relevancy (assessing how relevant the retrieved context is to the question). Both evaluators use intermediate outputs within the trace to calculate the final score.
You can now view traces directly in the playground. This feature enables you to debug your application while configuring it—for example, by examining the prompts sent to the LLM or reviewing intermediate outputs.
note
Both features are available exclusively in the cloud and enterprise versions of agenta.
Evaluators now can access all columns in the test set. Previously, you were limited to using only the correct_answer column for the ground truth / reference answer in evaluation.
Now you can configure your evaluator to use any column in the test set as the ground truth. To do that, open the collapsable Advanced Settings when configuring the evaluator, and define the Expected Answer Column to the name of the columns containing the reference answer you want to use.
In addition to this:
We've upgraded the SDK to pydantic v2.
We have improved by 10x the speed for the get config endpoint
We've improved the workflow for adding outputs to a dataset in the playground. In the past, you had to select the name of the test set each time. Now, the last used test set is selected by default..
We have significantly improved the debugging experience when creating applications from code. Now, if an application fails, you can view the logs to understand the reason behind the failure.
We moved the copy message button in the playground to the output text area.
We now hide the cost and usage in the playground when they aren't specified
We've made improvements to error messages in the playground
Bug Fixes
Fixed the order of the arguments when running a custom code evaluator
Fixed the timestamp in the Testset view (previous stamps was droppping the trailing 0)
Fixed the creation of application from code in the self-hosted version when using Windows
We've introduced a feature that allows you to use Agenta as a prompt registry or management system. In the deployment view, we now provide an endpoint to directly fetch the latest version of your prompt. Here is how it looks like:
from agenta import Agenta agenta = Agenta() config = agenta.get_config(base_id="xxxxx", environment="production", cache_timeout=200) # Fetches the configuration with caching
Previously, publishing a variant from the playground to an environment was a manual process., from now on we are publishing by default to the production environment.
The total cost of an evaluation is now displayed in the evaluation table. This allows you to understand how much evaluations are costing you and track your expenses.
Bug Fixes
Fixed sidebar focus in automatic evaluation results view
Fix the incorrect URLs shown when running agenta variant serve
You can now monitor your application usage in production. We've added a new observability feature (currently in beta), which allows you to:
Monitor cost, latency, and the number of calls to your applications in real-time.
View the logs of your LLM calls, including inputs, outputs, and used configurations. You can also add any interesting logs to your test set.
Trace your more complex LLM applications to understand the logic within and debug it.
As of now, all new applications created will include observability by default. We are working towards a GA version in the next weeks, which will be scalable and better integrated with your applications. We will also be adding tutorials and documentation about it.
Find examples of LLM apps created from code with observability here.
We've introduced the feature to version prompts, allowing you to track changes made by the team and revert to previous versions. To view the change history of the configuration, click on the sign in the playground to access all previous versions.
v0.9.1
We have added a new evaluator to match JSON fields and added the possiblity to use other columns in the test set other than the correct_answer column as the ground truth.
Up until know, we required users to use our OpenAI API key when using cloud. Starting now, you can use your own API key for any new application you create.
We've spent the past month re-engineering our evaluation workflow. Here's what's new:
Running Evaluations
Simultaneous Evaluations: You can now run multiple evaluations for different app variants and evaluators concurrently.
Rate Limit Parameters: Specify these during evaluations and reattempts to ensure reliable results without exceeding open AI rate limits.
Reusable Evaluators: Configure evaluators such as similarity match, regex match, or AI critique and use them across multiple evaluations.
Evaluation Reports
Dashboard Improvements: We've upgraded our dashboard interface to better display evaluation results. You can now filter and sort results by evaluator, test set, and outcomes.
Comparative Analysis: Select multiple evaluation runs and view the results of various LLM applications side-by-side.
This necessitated modifications to the SDK. Now, the LLM application API returns a JSON instead of a string. The JSON includes the output message, usage details, and cost:
We rewrote most of Agenta's frontend. You'll see much faster speeds when you create prompts or use the playground.
We also made many improvements and fixed bugs:
LLM-as-a-judge now uses double curly braces {{}} instead of single curly braces { and }. This matches how normal prompts work. Old LLM-as-a-judge prompts with single curly braces still work. We updated the LLM-as-a-judge playground to make editing prompts easier.
We spent the past months rethinking how evaluation should work. Today we're announcing one of the first big improvements.
The fastest teams building LLM apps were using human evaluation to check their outputs before going live. Agenta was helping them do this in minutes.
But we also saw that they were limited. You could only score the outputs with one metric.
That's why we rebuilt the human evaluation workflow.
Now you can set multiple evaluators and metrics and use them to score the outputs. This lets you evaluate the same output on different metrics like relevance or completeness. You can also create binary, numerical scores, or even use strings for comments or expected answer.
This unlocks a whole new set of use cases:
Compare your prompts on multiple metrics and understand where you can improve.
Turn your annotations into test sets and use them in prompt engineering. For instance, you can add comments that help you later in improve your prompts.
Use human evaluation to bootstrap automatic evaluation. You can annotate your outputs with the expected answer or a rubic, then use it to set up an automatic evaluation.
Watch the video below and read the post for more details. Or check out the docs to learn how to use the new human evaluation workflow.
The JSON editor now provides clearer error messages and improved editing functionality. We've fixed issues with error display that previously made it difficult to debug JSON configuration problems.
Undo Support with Ctrl+Z
You can now use Ctrl+Z (or Cmd+Z on Mac) to undo changes in the JSON editor, making it much easier to iterate on complex JSON configurations without fear of losing your work.
Bug Fix: JSON Field Order Preservation
The structured output JSON field order is now preserved throughout the system. This is crucial when working with LLMs that are sensitive to the ordering of JSON fields in their responses.
Previously, JSON objects might have their field order changed during processing, which could affect LLM behavior and evaluation consistency. Now, the exact order you define is maintained across all operations.
Agenta now supports images in the playground, test sets, and evaluations. This enables a systematic workflow for developing and testing applications that use vision models.
New Features:
Image Support in Playground: Add images directly to your prompts when experimenting in the playground.
Multi-modal Test Sets: Create and manage test sets that include image inputs alongside text.
Image-based Evaluations: Run evaluations on prompts designed to process images, allowing for systematic comparison of different prompt versions or models.
We've made significant improvements across Agenta with a major documentation overhaul, new model support, self-hosting enhancements, and UI improvements.
Revamped Prompt Engineering Documentation:
We've completely rewritten our prompt management and prompt engineering documentation.
Start exploring the new documentation in our updated Quick Start Guide.
New Model Support:
Our platform now supports several new LLM models:
Google's Gemini 2.5 Pro and Flash
Alibaba Cloud's Qwen 3
OpenAI's GPT-4.1
These models are available in both the playground and through the API.
Playground Enhancements:
We've added a draft state to the playground, providing a better editing experience. Changes are now clearly marked as drafts until committed.
Self-Hosting Improvements:
We've significantly simplified the self-hosting experience by changing how environment variables are handled in the frontend:
No more rebuilding images to change ports or domains
Dynamic configuration through environment variables at runtime
