Test and score my agents
I want to test how well my AI behaves before I ship it. Here's a stack that does this job, and a worked example of it in use.
A data scientist pairs with a coding agent inside notebooks
A data scientist is doing analysis and building with AI. Their daily stack is deliberately small: a coding agent and a notebook.
They pair with a coding agent — something like OpenCode at home or OpenAI Codex at work — inside a notebook environment such as Marimo. The notebook is the canvas where the analysis lives; the coding agent writes the code that runs in it, cell by cell, while the data scientist keeps judgment over the method. When a model is involved and they need to trust it, they reach for an evaluation harness: run the model against test cases, score the outputs, and treat “good” as a number they can track instead of a feeling.
The discipline this case study carries, drawn from sessions on agentic data science, is that the notebook keeps the work reproducible and the eval keeps it honest. The coding agent handles the typing; the data scientist owns the question and the verdict.
Sources
- Pair-programming agentic data science in Marimo (
raw/0027-ericma-marimo-pair-agentic-data-science.md) - Agentic data science, a Bayesian example (
raw/0025-ericma-agentic-data-science-bayesian.md) - Hamel Husain on evaluations (
raw/0021-hamel-evals.md)
The stack
Click a box to filter the list to just the tools that fill it.
flowchart LR
app(("Your agent"))
eval["Evaluation"]
app -->|is tested by| eval Evaluation
Tools that run your agent against tests and score the results.
- AgentOps — Observability built specifically for AI agents.
- Arize Phoenix — Open-source LLM tracing and eval.
- Braintrust — Evals and prompt playground for serious teams.
- Comet Opik — Open-source eval and tracing from Comet.
- DeepEval (Confident AI) — Open-source unit tests for LLMs.
- Galileo — Guardrails and evaluation for production LLMs.
- Laminar — Open-source observability and eval for AI agents.
- Langfuse — The most-used open-source LLM observability tool.
- LangSmith — LangChain's observability and eval platform.
- Latitude — Open-source prompt management and eval.
- Maxim AI — Evaluation and simulation for AI agents.
- Patronus AI — Automated evaluation and guardrails for LLMs.
- promptfoo — Test and red-team LLM apps, prompts, and agents.
- Weights and Biases Weave — LLM tracing and eval inside W&B.