I built labelswap so I stop paying a frontier model to pick a label

A support ticket comes in. A frontier model picks one of twenty labels, and I pay for the call every time. That is the tax on every agent workflow that classifies things.

Two Amazon engineers replaced Claude Sonnet 4.6 with Strands Decider to label internal tickets. They tuned a confidence cutoff, ran both side by side, and ended up about 44 times cheaper per ticket. labelswap is my simpler, clean-room take: a Naive Bayes classifier with a confidence gate, not a decision model like Strands Decider. It learns from labels my LLM already assigned, answers the confident cases locally in microseconds, and sends the rest back.

The pattern is the point: a cheap classifier in front of the expensive model, and you only pay for the calls that need judgement.

Here is how I am testing it:

1. Clone the repo and run pip install -e ..
2. Train on a labeled log with a target agreement rate.
3. Run labelswap shadow next to fresh LLM labels.
4. Switch to labelswap decide once agreement holds.

On a made-up demo log, it answered 136 of 141 tickets locally and matched the LLM 98.5% of the time. The demo is easy, so real logs will see less. Naive Bayes can be confidently wrong — one demo mistake scored 1.0 — so shadow mode comes first.

Takeaway: let the expensive model teach a cheap one, then check it.

Coverage: Hacker News.