Framer AI AgentsDesign and publish professional sites with AI
Promoted
oqoqo
Build evals and custom benchmarks for real-world tasks
663 followers
Build evals and custom benchmarks for real-world tasks
663 followers
Run eval experiments at scale in realistic environments. Define custom task sets to build your private benchmarks, measure how well agents can use any product, and find best models for your use cases. Generate dynamic insights to detect frictions in product interfaces or token inefficiencies.
Free Options
Launch Team / Built With
oqoqo
Hey Product Hunters, I’m Haritha, co-founder of Oqoqo
Every week there is a new model launch and yet another benchmark released in the wild. But they do not help product builders evaluate how well their products can be discovered and used by these agents and models, or talk about actual tasks their users would perform. Most benchmarks today exist in curated environments and do not translate well to the real world.
We built Oqoqo to bridge this gap. Oqoqo makes it super simple to build realistic evals and custom benchmarks for tasks users actually care about.
With Oqoqo, you can define tasks as simple as a prompt your user might give to an agent e.g. “integrate supabase to my webapp to store user sign ups”, provide what you want to test for e.g. Supabase SDK, API, CLI etc. and define what success looks like e.g. “must set up RLS”.
We handle the rest. Our infrastructure spins up isolated sandboxes, executes the tasks against agents of your choice, catalogs every single step the agents take including tool calls, retries, discovery loops etc, and documents token consumption, cost, along with evaluating success/failure based on your success criteria.
With Oqoqo you can:
Reliably measure how agent friendly your product surfaces are against Codex, Claude Code, OpenClaw, Hermes, Pi, Opencode, Cursor, GitHub Copilot
Regression test MCP, CLI, skills, SDK, and any agent facing interface (we are continuously using Oqoqo to dogfood and improve our own MCP/CLI)
Create and share custom benchmarks for how agents discover and use your product
Compare models and harnesses for domain specific tasks
See whether new versions improve agent experience
We built Oqoqo for teams building products that agents want to use, and for teams putting agents into day to day work.
And the best thing? Your agent can handle the setup for you ✨, try it out for free today: https://oqoqo.ai/
We would love to learn what kind of experiments you would like to run and what questions you have about agent interactions and agent experience.
@margharitha Great Work!
Evaluation becomes a major challenge once AI systems move beyond demos. What experience pushed you toward building a dedicated platform for this problem?
oqoqo
@noahanderson Most eval platforms today seem like they need a data scientist to operate and yet don't reflect the real systems (dependencies, complex file context etc) needed for it. Oqoqo comes from the effort of trying to make it approachable for people to do evals that matter to them in their day to day without losing the nuance.
the realistic environments part is the right fight. the thing id watch next is eval rot, a case written six months ago measures the world as it was the day someone wrote it, and a suite that stops failing looks exactly the same as a product that got good. the number id surface is what share of cases have ever failed, because the ones that never have arent tests, theyre decoration, and they pile up until the green means nothing
oqoqo
@jernej_jan_kocica that's a great point. definitely worth exploring how to flag outdated eval sets. thanks jernej
Macaly
evals on real world tasks is the actual bottleneck rn, not another leaderboard 🙌 which use cases seeing the most pull so far?
oqoqo
@petrkovacik we have found testing MCPs and CLIs to be a real painful use case. Also, comparing different harnesses and models for personal tasks seem to be really interesting to people.
the same task rarely takes the same path twice with an agent, different tool call order, different retries. how are you keeping the scoring stable run over run so a benchmark result doesn't just become noise from agent nondeterminism
oqoqo
@sabber_ahamed Great question! One of the key reasons you should do such evals is exactly this non determinism. In a deterministic system, once you write unit tests, you are good. But with agents interacting with your surfaces, this changes significantly. One way to bring back some predictability is by running multiple trials so you have statistical significance to know what is the shared behavior across runs.
The project description is interesting, but I couldn't actually test it. There’s a demo on the homepage, so I entered my own task and submitted it. The behavior was a bit strange: at first, it wasn't clear whether anything was happening, and then a results page opened, but it didn't contain the results for my task. So I couldn't figure out whether this is a real interface where I can submit my own tasks, or whether it's a recorded/demo flow that always shows the same thing.
oqoqo
@natalia_iankovych Hey there, thanks for your interest! The demo on the landing page is just an interactive video. If you want to actually use the platform, feel free to hit "Get started" and "Sign up" at which point you'll be taken to "https://app.oqoqo.ai" where you can connect your model providers and actually create tasks and run experiments. The free tier gets you a generous 100 runs that renew every month and you don't need a credit card either! Learn more from our docs here: https://docs.oqoqo.ai/quickstart.
Let us know if this clarifies things and if you need any further help, we'll be happy to chat.
@raokarthik99 But I was able to enter my own task in the input field and submit it, which makes the demo somewhat misleading
Cloud World Model
Congrats @margharitha Haritha and team on #1 Product of the Day! 🔥
I tested Oqoqo with a pretty tough @Cloud World Modelreliability task, and the most interesting part was watching the agents decide how to solve it.
Same task, but one used the CWM MCP while the Raw Agent basically said, “Nah, I got this” 😂 and built its own environment, assumptions, and logic. The Raw Agent was non-deterministic. Every time, it built a simulation, it built something different and used a different methodology each time.
Even the one that used CWM MCP, injected its own logic or assumptions at times.
This is very important for MCP operators. I used @Grok CLI (Unofficial) and other products and it just uses the MCP server where as Oqoqo MCP integration seems to throw in own logic or assumptions at times. I’d love to see Oqoqo make that behavior measurable: Did the agent actually use the product as intended, or did it go off and do its own thing?
That feels like a really valuable dimension of agent evaluation. Congrats again! 👏🏾
oqoqo
@mathsociety Thanks Kevin for trying out Oqoqo. Did the agent actually use the product as intended, or did it go off and do its own thing? -> Did you find the ability to add specific rubrics like 'should have used CWM MCP in a certain way' helpful to solve this?
Cloud World Model
@margharitha Thanks Haritha! The rubrics definitely help, but I think what I ran into is slightly different. I can add a rubric saying “use CWM MCP,” but the agent can satisfy that and still inject its own assumptions/logic around the MCP results.
What was interesting was discovering behavior I didn’t know to write a rubric for beforehand. The Raw Agent went completely DIY 😂, while the MCP agent used CWM but occasionally added its own interpretation.
I’d love a way to measure something like tool fidelity / agent deviation automatically, how much of the final result came from the product/tool versus logic introduced by the agent. That would be really useful for MCP builders.
oqoqo
@mathsociety that makes a ton of sense, and great feedback. We do have a 'run profile' feature where we were trying to capture some of this but really interesting angle on agent deviation.