I Tested Jev on 1,000 IT Tickets. The Dataset Was the Weak Link.

Out of 1,000 IT tickets, Jev made 4 routing mistakes that nobody would have caught. Four. Everything else was either routed to the right place or flagged for a person to take a second look.

That is the short version. The long version is a little more interesting, and honestly a little messier, so here it is.

What I tested

Jev is the routing layer I built for Salesforce. A ticket comes in, Jev reads it, picks the queue it belongs in, and attaches a confidence score to that pick. Simple enough. The question I wanted answered was just as simple: if I let Jev route tickets on its own, how often does it send one to the wrong place?

To find out, I pulled 1,000 IT tickets from a public Kaggle dataset and had Jev sort them into 8 queues. The dataset comes with labels already attached, so I had an answer key to grade against. At least that is what I thought going in.

The raw score: 67.6%

Graded straight against Kaggle’s labels, Jev matched 676 of the 1,000 tickets. That is 67.6%, and I will be the first to say it is not a number you would ever ship.

But once I started actually reading through the misses, a pattern showed up fast. A lot of them were not Jev being wrong – they were Kaggle being wrong. The label on the ticket just did not match what the ticket was about. Think of a teacher grading an exam with an answer key that has mistakes in it. A student could get the question right and still lose the points, and no amount of studying fixes that. That is what was happening here.

So 67.6% is not really a measure of Jev. It is a measure of how often Jev agrees with a noisy dataset. Which raises the next question: how many of those “misses” were actually misses?

Re-checking the 324 disagreements

Jev and Kaggle disagreed on 324 tickets. I had those re-reviewed blind by an AI model (Claude). It saw the ticket and the two queue options, but it did not know which one was Jev’s pick. No thumb on the scale, basically.

Here is how it shook out:

That flips the whole story. Of the 324 “misses,” more than half were Jev getting it right while the answer key got it wrong. Only 56 were real mistakes.

Add it all back up and Jev lands at about 90% accuracy overall. The range is 87% to 94%, depending on how you count those 76 judgment calls. Count every one of them against Jev and you get the low end, count every one for Jev and you get the high end. The truth is somewhere in the middle.

The part that actually matters: the 0.90 cutoff

90% accuracy is good, no question. But would you let a model blindly route every single ticket at 90%? I would not. One in ten tickets landing in the wrong queue is a real problem for the person on the other end waiting for help.

This is where the confidence score earns its keep. Jev gives a score with every pick, and I set the cutoff at 0.90. At or above it, the ticket routes itself. Below it, a person takes a look. I would compare it to a new hire on the help desk – you do not need them to know everything on day one, you need them to know when to go ask someone. That is exactly what the cutoff does.

At that cutoff:

• 57% of tickets (570) were auto-routed, at about 97% accuracy. Only 4 of the 570 were clear mistakes. That is 0.7%.
• 43% went to a person. For these, Jev’s first guess was wrong 12–26% of the time, which is exactly the pile you want a human looking at.

The more interesting question is where the 56 real mistakes ended up:

52 of the 56 mistakes, 93%, fell below the cutoff. Jev was wrong, sure, but it also knew it was not sure – and that is what kept those tickets away from auto-routing.

At the end of the day, across all 1,000 tickets, 4 would have gone to the wrong queue without anyone catching it. That is the number that matters.

Why this setup is optimal

Accuracy is only half of it. A router can be right 99% of the time and still be the wrong choice if it is slow or costs a fortune to run on every ticket. Jev checks the other boxes too.

• It is cheap. Jev is a small, inexpensive model. Routing is a high-volume, repetitive job, so the cost per ticket matters more than anything. You do not need the most powerful model on the market to decide whether a ticket is a password reset or a hardware issue – that is like hiring a surgeon to put on a band-aid.
• It is fast. Jev routes tickets almost as soon as they come in. No sitting in a general inbox waiting for someone to get to it, the ticket lands in the right queue and the right team can start on it.
• It takes routing off the service desk’s plate. Right now someone on the desk has to read every ticket, decide where it goes, and keep the queue moving. That is time spent sorting instead of solving. With Jev, more than half of tickets never need a person to touch the routing at all, and for the rest, the person is just confirming or correcting Jev’s suggestion instead of starting from scratch.

Put those together and the desk gets back the hours it was spending on triage and queue management, without paying a premium for it. Cheap, fast, and accurate where it counts – that is the combination you want.

Caveats

I would rather be upfront about where these numbers could be off than have someone else point it out.

• The re-review was done by an AI, not a person. Claude judged the 324 disagreements blind, but nobody has hand-checked its calls yet.
• Only the disagreements were re-reviewed. The 676 tickets where Jev and Kaggle agreed were taken as correct. If both of them got the same ticket wrong, I would not know.
• The dataset is rough – very rough. Kaggle’s ticket text is heavily stripped down, with no stopwords and no names. Real tickets read like real sentences, so they should be easier to route, not harder. If anything, testing on a dataset this poor makes the result more encouraging, not less.

The takeaway

The headline accuracy number was never really the point. What matters is whether Jev can tell when it is guessing. On this test it could. When it was confident, it was right 97% of the time, and when it was wrong, it usually said so.

That is what makes auto-routing safe. More than half the queue handles itself, the people on the team only see the tickets that actually need them, and the miss rate on everything that skips a human is under 1%. Not because the model is perfect, not because the dataset was easy, but because the model knows its own limits and the cutoff respects them.

Next step is putting Jev on real tickets and having a person verify the results.