We built and open-sourced Ragas in mid 2023 with the aim of building an evaluation tool that would help make evaluation of RAG applications easier. At the time, RAGs were among the most widely deployed LLM applications, and ensuring their evaluation was crucial. Fast forward one year, LLM applications are going much beyond RAG and Ragas is also evolving to accommodate this need for evaluating LLM applications like agentic workflows. Ragas 0.2 is our first step in this direction. Our goal is to build a supercharged workflow for evaluating LLM applications.
What’s new?
Few highlights on v0.2 release
- New dataset interface: supports preparing dataset for evaluating single shot, conversational and agentic workflows.
- New list of available metrics: contains LLM and non LLM based metrics evaluating various applications from RAG to Agentic workflows.
- New test data synthesiser: improved version of test data generator from documents that supports creating custom scenarios for evaluating RAG, higher quality QAs, lower cost test and reusability.
Checkout the migration guide to start using Ragas 0.2.
What’s next?
For the immediate future, we will be working on
- Methods to improve the quality and alignment of LLM based scoring
- Improving the coverage and quality of synthetic test data generation
- Reproducible evaluation workflows for agentic/tool-use applications.
We’ll soon be discussing, preparing, and sharing our concrete roadmap for the next three months
Community
We believe our community is the cornerstone of ragas’ success, and we want to extend our heartfelt thanks to every contributor who has played a role in improving it. If you contributed a pull request before our 0.2 release, we’d love to send you a token of appreciation. Please fill out this form if you’d like to receive it. We’re committed to growing a collaborative community where everyone can share ideas and work together to make ragas even better. If you’re already not in our discord, consider joining discord, and following us on X and LinkedIn.