Gave the business visibility into how users respond to AI-generated legal edits, with acceptance tracking, per-tenant cost, and a custom trace export pipeline.
LLM observability covered the technical basics but missed the product and business questions. Nobody could see what share of users accepted, rejected, or rewrote AI-suggested contract edits, so the business couldn't tell whether the LLM features were actually helping. Cost per tenant was unknown, and traces were hard to analyze outside the LangSmith UI.
Approach
Tracked user responses to AI edits. I instrumented the frontend and backend to record accept, reject, and rewrite events in LangSmith, attached to the trace that produced the suggestion. Each piece of feedback links to the exact prompt, model, and output behind it, without duplicating trace data.
Built a custom exporter from LangSmith to AWS to preprocess and persist traces, so they can be analyzed freely and kept beyond the vendor's retention.
Made cost per tenant visible by aggregating token usage from the exported traces for each customer.
Feedback is attached to the trace that produced the suggestion, then exported to AWS for product and cost analysis
Impact
The customer success team got a clear view of product adoption, including whether AI suggestions were getting better or worse over time. That view informed product decisions.
Usage and cost data per tenant informed pricing adjustments.
