Turned a manual, local eval harness into a daily automated pipeline with legal-domain evals, full data lineage, and Grafana dashboards for the whole team.
The agent evaluation harness was run by hand on local machines. Results weren't stored, there were no dashboards, and nobody could see how model or prompt changes affected quality over time. The existing evals were generic and missed the failure modes that matter in legal documents.
Approach
Legal-domain evaluations. I built domain-specific evals together with legal experts, targeting the regressions that matter to lawyers rather than generic LLM benchmarks.
Automated runs. A daily GitHub Actions workflow, integrated with AWS, runs the eval harness without anyone having to start it.
Serverless processing.AWS Lambda jobs process the results and store them in S3, together with the metadata needed to reproduce each run. Athena makes the results queryable with SQL.
Dashboards.Grafana shows results over time for engineers, product, and the legal team.
Daily eval pipeline from GitHub Actions through AWS to Grafana
Impact
Evals went from manual and local to automated daily runs.
Full data lineage and reproducibility for every eval run.
A historical, visual record of quality changes that technical and non-technical teams can both use.
