Nimble
SOTA Decision Model
101 followers
SOTA Decision Model
101 followers
Today, we are announcing Nimble-v4 and Nimble-v4-max which are pushing the frontier of decision models, outperforming Jev and other variants. On Decision Index public scores, our nimble-v4-max model and nimble-v4 models score +10.5 and +5.74 against Jev 1.13, making nimble-v4-max a top performing model. Beyond benchmarks, we have done a lot of internal evals, and see big improvements: 43% fewer mistakes than Jev, +50pp on model routing decisions, and so on. Try for free: console.bespokelabs.ai
We released Nimble-v1 the same week as Jev, which has spurred a lot of activity in the community. For the past few weeks, we have been making meaningful advancements in data curation and modeling techniques, and put our platform to work, resulting in SOTA decision models.
Try it for free: console.bespokelabs.ai
We’d love to hear which decision in your stack you’d hand to Nimble first, and where it falls short. We’ll be here all day answering questions. 🙏
Congrats on the launch! Curious about confidence calibration, are the output probabilities calibrated against empirical accuracy, or are they derived directly from softmax scores? Any plans to support temperature scaling?
@arielrg Great Q. We are working on supporting calibration against empirical accuracy. Will share more on that soon!
@pratikd15 Awesome, looking forward to future updates!
@pratikd15 impressive to see Nimble pushing the boundaries of decision models, How does it compare with other leading models in real-world tasks ?
@dipanshu_kushwaha5On public benchmarks, nimble-v4-max scores +10.5 over Jev 1.13 on Decision Index. Our internal evals are built around real-world tasks, and there we see 43% fewer mistakes than Jev and +50pp on model routing decisions. Do test it with your own data - the console has free credits, so you can run your own decisions through it and compare.
Can I set my own threshold for when a person should step in? Good luck today !!
Congratulations on the launch, team! Very impressive stats.
@arjun_rakesh_k_o Thank you!
Congrats on the launch team!!!
@zahle_khan Thank you!