Support operations leader who builds with AI. I run a globally distributed technical support team and put AI into production support workflows. This is where the tools from that work, and a few from outside it, go public.
Everything here is open source. Anything marked Live runs in your browser with nothing to install.
The measuring is the part that usually gets skipped.
Your AI vendor reports 85% deflection. This audits what that number survives. It re-judges every AI-closed conversation from the transcript alone, then recomputes the rate crediting only closures that hold up. On a 40-conversation labeled set, 80% claimed came out to 55% audited.
Test a vendor's AI support agent against your own rules before it talks to your customers. It grades what the agent did against what it said, so a reply promising a person with no handoff behind it is its own failure. A wrong call on a rule that conflicts or has a gap goes to the rules owner, not the vendor. Back-tested on a real vendor's staging agent, it matched hand grading on 35 of 39 tests.
A scheduled agent that hits a permission prompt can't answer it, so it skips the write and still reports success. A nightly check that ignores what each agent says about itself and rules on whether the file it was supposed to produce actually moved. Misses get a named failure signature, and a folder it can't read is never called a failure.
What is each scheduled agent actually allowed to touch? It reads the permission config the agents run under and each agent's instructions, then diffs both against a twelve-rule acceptable-use policy. Every capability is labeled declared or inferred. The first real run found 51 of 55 agents working from the home folder, with every credential file in reach.
Agent skills are prompt-matched, so a model upgrade can stop one from firing without an error anywhere. Static checks plus live dispatch tests covering every installed skill, run under a tool denylist so skills that reach Salesforce and Gmail can't touch either. The one recorded live run, on the 23 prompts that existed then, got 21 right.
Does a second agent pay for itself? Generalist, parallel specialists, and generalist plus QA reviewer, measured on accuracy, cost per ticket, and latency. Specialists took routing from 83% to 97% and left severity flat; the QA layer lifted severity from 70% to 80% at $0.60 per net correction.
The sequel to agent-ops-bench: a challenger that can only object, rules that only raise urgency, and a route to a person. Six configurations on 60 cases I labeled by hand landed within noise of each other, and Haiku alone matched the full system at 1/40 the cost. The bigger gap was between the written taxonomy and my own labels, and adjudicating it is what moved priority accuracy from 60% to 88%.
The eval machinery from these projects, pulled out into something reusable. Blind classification, confusion matrices, ordinal miss-direction splits, and confusable-pattern trap audits. It's the scoring layer under deflection-audit and the skill harness.
An LLM triage classifier with a real eval harness. Severity and routing for IoT support tickets, scored against a 60-ticket labeled set with under- versus over-triage analysis. It caught all 12 critical tickets.
Production knowledge from running a support org, encoded into something another team can use.
When a support copilot gives a wrong answer, the root cause is often the article, not the model. Paste a help-center article and get an AI-readiness grade across structure, answer clarity, self-containment, machine readability, and language quality, with a concrete fix for each finding.
Read-only Salesforce case operations as MCP tools an agent can call. SOQL, case lookups, queue volumes, and SLA milestone risk. Auth stays in the sf CLI keychain, read-only is enforced by construction, and it runs against a live org daily.
A thermostat export is thousands of telemetry pings, and about 1% of them are what anyone asked about. Drop one in and get the setpoint changes, or a month-by-month timeline with state and target carried forward. It handles the quirks you only learn by misreading these files, and it runs in the browser, so nothing gets uploaded.
The expensive capacity question isn't next week. It's who's on the bench in March, and whether February's gap meant opening a req last month. A rolling 26-week forecast for a services team: bench by engineer, skill-mix gaps, and the date each req has to open once hiring and ramp are counted. On the sample team, the AI-agents gap lands in February and the req opens in November.
Job discovery and resume tailoring built on Claude Code. It polls 50+ company job boards, and a calibration audit re-derives every score from the rubric that produced it, flagging decisions made under rules that have since changed.
Same habits, different problems.
MQD Runway: project your year-end Delta MQDs and price out what closes the gap to each Medallion tier. Every program rule is editable data, because loyalty programs change their math and hardcoded calculators go stale.
A native-plant planner for the Georgia Piedmont that solves for unbroken bloom. It keeps something flowering from March through November and treats heavy clay as a real constraint rather than a footnote.
Evidence grading for research peptides. Per-claim provenance and a fixed rubric, published as one standalone record per compound.