Where the agent's autonomy ends

A support agent earns its keep by answering the simple cases on its own and staying out of the ones where a mistake is expensive. A complaint, a refund request, or a question the knowledge base has no answer for has to go to a human, even when the model claims to be confident. So this project isn't about generation quality; it's about the boundary: where the agent is allowed to act, who checks that, and what happens when something breaks.

Scope: the service runs in Russia (foreign LLM APIs aren't reachable, personal data handling follows 152-FZ), a budget of $0.02 per ticket, a human kept in the loop, and every decision traceable.

What got built: a design document, requirements, use cases, a decision table, an ERD, API contracts, 12 ADRs, a test strategy, and SLIs. Then an implementation built from it: FastAPI, LangGraph, Postgres with pgvector, RabbitMQ, an operator console in React. And measurements that checked both the document and the code.

The decisions everything else rests on

What the operator sees

The escalation reaches the operator carrying exactly what the agent based its decision on: the conversation, the rule that fired, the classification with its confidence source, and the documents search found, in the version the agent actually saw. The draft reply can be sent as is, edited, or replaced from scratch; the edit is saved alongside the draft for quality review.

The demo stand runs a baseline: a dictionary classifier and a templated draft. It comes up with one command:

What the measurements found

The most valuable parts of this project are the places where a measurement overturned a decision that looked correct on paper.

The load test found three capacity defects. At 50 concurrent tickets, the auto-reply p95 was 12.98s against an 8s budget. The causes weren't in the model:

• lru_cache isn't thread-safe: the first wave of requests created a database engine per thread, and connections climbed to 56;
• a request held its database connection for the entire time it waited on the LLM;
• anyio's default thread pool is 40, against 50 concurrent tickets.

After the fixes: 6.30s at 50 concurrent and 6.41s at 100, with no more than 22 database connections.

ADR-010 set the rule in advance: if Recall@5 falls below 0.9, turn on hybrid search. Recall came in at 0.73, so hybrid search was turned on: Postgres full-text search with Russian morphology, plus vectors, merged by rank (RRF). Confidence stayed cosine-based: the threshold is calibrated on that score, not on the fused one.

Stop-word filtering added another 2 points, but its threshold would have had to be tuned on the same set the measurement runs against. Rejected as fitting the test.

Building the implementation from the document also became a way of checking it. Discrepancies found and closed:

• the schema was described as "designed for hybrid search from the start," though it had no full-text index;
• the code referenced a runbook twice that didn't exist;
• the "end-to-end trace_id" was actually different across the agent, the error, and the queue event;
• escalation on LLM unavailability bypassed the agent entirely, with no log entry, contradicting the traceability requirement;
• workers crashed when Postgres restarted, and a connection through the Docker proxy hung indefinitely with no timeout.

Non-functional requirements

The report is assembled by a script from run artifacts; no number is entered by hand.

What this project doesn't prove

• The golden set is synthetic. The dictionary classifier the cross-check relies on was written against those same phrasings, so real tickets will show more disagreements and fewer auto-replies.
• The load test simulates LLM latency. Against the real provider with no load, processing p95 is 1.9s; it hasn't been checked under load or against the provider's own rate limits.
• There were no real tickets or operators: the share of operator edits and the quality of the drafts haven't been measured.

Each of these limits is recorded in its corresponding ADR and in the NFR report, not just here.

Code, the design document, and run artifacts