Launching today
oqoqo
Build evals and custom benchmarks for real-world tasks
488 followers
Build evals and custom benchmarks for real-world tasks
488 followers
Run eval experiments at scale in realistic environments. Define custom task sets to build your private benchmarks, measure how well agents can use any product, and find best models for your use cases. Generate dynamic insights to detect frictions in product interfaces or token inefficiencies.
Free Options
Launch Team / Built With
oqoqo
Hey Product Hunters, I’m Haritha, co-founder of Oqoqo
Every week there is a new model launch and yet another benchmark released in the wild. But they do not help product builders evaluate how well their products can be discovered and used by these agents and models, or talk about actual tasks their users would perform. Most benchmarks today exist in curated environments and do not translate well to the real world.
We built Oqoqo to bridge this gap. Oqoqo makes it super simple to build realistic evals and custom benchmarks for tasks users actually care about.
With Oqoqo, you can define tasks as simple as a prompt your user might give to an agent e.g. “integrate supabase to my webapp to store user sign ups”, provide what you want to test for e.g. Supabase SDK, API, CLI etc. and define what success looks like e.g. “must set up RLS”.
We handle the rest. Our infrastructure spins up isolated sandboxes, executes the tasks against agents of your choice, catalogs every single step the agents take including tool calls, retries, discovery loops etc, and documents token consumption, cost, along with evaluating success/failure based on your success criteria.
With Oqoqo you can:
Reliably measure how agent friendly your product surfaces are against Codex, Claude Code, OpenClaw, Hermes, Pi, Opencode, Cursor, GitHub Copilot
Regression test MCP, CLI, skills, SDK, and any agent facing interface (we are continuously using Oqoqo to dogfood and improve our own MCP/CLI)
Create and share custom benchmarks for how agents discover and use your product
Compare models and harnesses for domain specific tasks
See whether new versions improve agent experience
We built Oqoqo for teams building products that agents want to use, and for teams putting agents into day to day work.
And the best thing? Your agent can handle the setup for you ✨, try it out for free today: https://oqoqo.ai/
We would love to learn what kind of experiments you would like to run and what questions you have about agent interactions and agent experience.
Evaluation becomes a major challenge once AI systems move beyond demos. What experience pushed you toward building a dedicated platform for this problem?
oqoqo
@noahanderson Most eval platforms today seem like they need a data scientist to operate and yet don't reflect the real systems (dependencies, complex file context etc) needed for it. Oqoqo comes from the effort of trying to make it approachable for people to do evals that matter to them in their day to day without losing the nuance.
the same task rarely takes the same path twice with an agent, different tool call order, different retries. how are you keeping the scoring stable run over run so a benchmark result doesn't just become noise from agent nondeterminism
oqoqo
@sabber_ahamed Great question! One of the key reasons you should do such evals is exactly this non determinism. In a deterministic system, once you write unit tests, you are good. But with agents interacting with your surfaces, this changes significantly. One way to bring back some predictability is by running multiple trials so you have statistical significance to know what is the shared behavior across runs.
Have you noticed big differences between Claude Code, Codex, Cursor, and Copilot when running the exact same real world task?
congrats @margharitha & team!
oqoqo
@hamza_afzal_butt definitely. There are a lot of cool insights you can find by doing such running against multiple agents. For example we have found that codex tends to spend a lot more time researching than implementing but ultimately finds the right answer whereas claude code tends to implement and iterate a lot more until it finds the right solution.
Bababot
Can teams compare different models on the exact same custom task set?
oqoqo
@aarav_pittman Yes, ofc! And it's not just for model comparison but you can compare any agent/harness. Even within a given harness, you can compare models and even within a model, you can compare even amongst different reasoning levels/efforts. An experiment is the same custom task set spanned across all these permutation/combinations.
The part eval systems often miss is recovery behavior: permission denial, stale credentials, partial side effects, and a rerun after failure. A benchmark that scores the happy path but not cleanup and recovery can reward an agent that looks finished while leaving the product in a worse state.
oqoqo
@krekeltronics Definitely! One thing we have found helpful is to add negative rubric criteria that makes sure we are measuring when things go wrong as well.
Serand
oqoqo
@rukhsar_amjad The criteria can be as elaborate/simple as you want, having criteria around quality definitely helps manage this.