About six years ago we built a ticket classification/routing model and pipeline [1]. The initial version was trained using SVMs, it took a couple of minutes to train, and the ticket classification was just a few milliseconds. It's so fascinating to watch how the field moved over to LLMs, and then back to faster/cheaper classifiers again.
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
armcat
At ~$21,700 saved per million tickets, saving millions annually would suggest at least 92 million tickets a year to reach $2 million saved. Is that about right?
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
Kaedon
LLM is a hammer that makes you see everything as an LLM usecase.
Recently people started equating all cognitive abilities to an LLM. Every foundational work that happened in ML and data science prior to LLm, are largely forgotten in the flood of LLM hype.
A lot of usecases would be far better off using some NLP or decision systems, or rule engines rather than paying for Opus. Big companies blindly use expensive frontier moels as a blanket solution irrespective of usecase fit, because they are chasing the goal of being an "AI-powered" company. LLM killed all othe simpler, specific solutions.
zkmon
One of the biggest issues with using a general LLM like Sonnet is calibration. I think Jev really helped bring this to light. With Sonnet, you get an answer and that's it. It's really not that great for classification, especially tagging.
I've trained a lot of NLI models, which do similar things to decision models. A 1B parameter encoder is about the same capability as a 2B LLM, but if you have multiple tags to check, it's not as efficient because you can't cache the bidirectional attention. On the other hand, they do seem to do a bit better on calibration.
This article is so confusing. For example, they say they tried the Strands Decider model and dismissed it for Gemma, then they continue to call the model they used Strands Decider? Did I miss something?
Gasp0de
Ok, but why did they chose Sonnet in the first place? Couldn't they have chosen Haiku or GPT Luna or maybe even some small cheap Chinese model?
psnehanshu
It's baffling to me that a company like Amazon would choose to use a LLM for a task as simple as ticket classification. Don't they have thousands of engineers and petabytes of data? They even have their Mechanical Turk program if they want a cheap labelling workforce.
Otterly99
Classifying into one of 20 groups? Did anyone try scikit learn?
3eb7988a1663
So why did they not even try Haiku 5.5?
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
origini
You can do this even more efficiently with the rust crate bonsai. You will have more control and you can create sceneries of logic. You can use incredibly cheap models this way and get way better results.
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
comments (10)
[1] "Linnaeus: A highly reusable and adaptable ML based log classification pipeline", https://arxiv.org/abs/2103.06927
NOTE: the work was actually accepted at the International Conference on Software Engineering (ICSE) in 2021.
armcat
I initially thought of tickets as just customer support, or internal JIRAs, but the article mentions alarms, security findings, and capacity increase requests, so that makes the scale feel more plausible. Still surprising so many tickets go through the system.
Kaedon
Recently people started equating all cognitive abilities to an LLM. Every foundational work that happened in ML and data science prior to LLm, are largely forgotten in the flood of LLM hype.
A lot of usecases would be far better off using some NLP or decision systems, or rule engines rather than paying for Opus. Big companies blindly use expensive frontier moels as a blanket solution irrespective of usecase fit, because they are chasing the goal of being an "AI-powered" company. LLM killed all othe simpler, specific solutions.
zkmon
I've trained a lot of NLI models, which do similar things to decision models. A 1B parameter encoder is about the same capability as a 2B LLM, but if you have multiple tags to check, it's not as efficient because you can't cache the bidirectional attention. On the other hand, they do seem to do a bit better on calibration.
I keep a leaderboard here: https://huggingface.co/spaces/dleemiller/open-nli-leaderboar...
deepsquirrelnet
Gasp0de
psnehanshu
Otterly99
3eb7988a1663
Why would Amazon go to another company to let them provide an adjusted decider model?
Amazon already has all these tickets labeld, they could just have build it themselves?
origini
I built a real time character story. In which players interact and integrate fictional characters. There is no delay in response and the characters are all in character. Oh and it’s all voice.
It’s shocking how effective this is.
tdhz77