Open source
AI evaluation toolkit
A set of open-source tools for testing AI systems: prompt quality, retrieval-augmented generation (RAG) quality, and agent behavior.
The Challenge
Most AI systems ship on a handful of manual spot checks. Without repeatable tests, nobody can tell whether a change made things better or worse, and problems show up in front of customers.
How It Works
1
Prompts are compared on a shared dataset.
Prompt variants run against the same test cases and are graded two ways: automated checks and an AI judge. Models are routed by task, so grading uses a stronger model than generation.
2
Retrieval is measured separately from answers.
RAG evaluation scores whether the right source material was found, using standard ranking metrics, then whether the answer used it. That separates retrieval failures from answer failures.
3
Agents are tested end to end.
An agent harness runs datasets through the agent, scores the results and reports them, with tests running automatically on every change.
Built With
Python, LiteLLM, Streamlit, pytest, GitHub Actions
What it's designed to do
- Turn "it seems to work" into measured results
- Catch regressions before they reach users
- Work across AI providers, so you're not locked in to one
Want something like this for your business?
Every build starts with an assessment, so we can adapt this pattern to how you actually work.