








In this time when we use LLM's for basically anything, what AI agents/tools would you want that you would actually pay for?
What are you spending a lot of money on and would probably save a lot of you time if you got the Agent out of box, or if configuration was much easier?
We wanted to come back and say thank you, properly.
Since launching here, Kopai crossed 150 users, and a lot of that traces directly back to this community. The upvotes mattered, but the comments mattered more: several of you asked sharp, specific questions that are already shaping what we build next.
Every team building agents rebuilds the same plumbing: a tool layer, a RAG pipeline, connectors to their data, and some orchestration glue to chain prompts together. It's weeks of undifferentiated work before you ship anything a user sees.
We are building to be that layer, managed:
Tools give your agent capabilities without writing the harness
RAG attach a knowledge base, retrieval handled
Connectors plug into the systems your agent needs to read and act on
Prompt orchestration chain, branch, and route across steps
Metering every run is measured, so billing and cost visibility are built in, per agent
the marketplace angle is what got me, being able to try out an agent someone else built and pay per message instead of committing to a subscription for something I'm not sure will be good. the eval/liability gating on published agents is more thought through than I expected for something this early
the transparency on whether an embedded agent updates live vs a frozen snapshot could be clearer up front, and I'd like to see more published agents in some of the more technical categories, still thin outside a few popular ones
looked at just building my own custom GPT or using a generic agent builder, but everything felt like it needed way more setup for a one-off use case. kopai's pay-per-message model made trying a few different specialist agents low risk
the part I'd want to understand before uploading my own knowledge base: when the agent gives someone a wrong or outdated answer under my name, who's actually on the hook for that, me or Kopai? per-hour consulting has a built-in correction loop, the client pushes back live and I clarify. per-message, the buyer just gets an answer and leaves, so a bad take can sit there generating "instant answers" indefinitely before anyone notices it's stale. is there a review or flagging loop on the expert's side, or does it rely entirely on buyers reporting bad responses after the fact?
Hi @galdayan !
This is honestly a really good question. Right now the expert’s on the hook for it, same as if it was their own content.
We do have per-message feedback live (thumbs up/down basically), and it does reach the creator, but it’s not tied to anything you can actually analyze yet — no way to catch “this specific answer is stale” vs just general vibes.
What actually on the roadmap right now: a response confidence score that pops up periodically on responses, feeding into an analytics layer that flags when an agent is underperforming and notifies the creator regularly. If in a case the score drops too low, the agent will get auto unpublished from the marketplace, until the creator fixes it and our evals re score it, and it crosses the threshold. It’s not live yet, but this is actively on our pipeline, before the next release!
Thanks for bringing this out! Would love you pressure test this. More feedback is always welcome!
You can reach out to me as well - swapnanil@usekopai.com
One more thing worth mentioning is our Eval layer (which I briefly mentioned earlier) inside of the agent builder, which is currently live. Before any agent goes live (or gets updated), it runs through an evaluation layer that scores it across several dimensions: system prompt quality, behavior/scope adherence, and safety (policy refusals + jailbreak resistance) are always checked, with knowledge base retrieval accuracy and tool-use correctness, when the agent actually has KB or any tools enabled. It's a gate, before any agent goes to the marketplace, without it just being a post-facto monitoring tool. Every time a creator changes their agent (new prompt, new knowledge base doc, config tweak, whatever), it automatically gets re-evaluated before the update is live for buyers. So it's not "publish once and hope", essentially every change re-triggers the check.
@swapnanil_ray that eval-gate answers most of my worry actually, re-triggering on every edit is the right call. one thing I'm still curious about - who sets the "too low" threshold for the auto-unpublish, and is that number visible to the expert before it happens? if it's a black box you'd get experts optimizing for the eval instead of the actual answer quality, which is its own failure mode.
@galdayan glad that cleared your worry! The publish gate is live today, and the threshold is 70 out of a full score of 100, its fixed, and not hidden anywhere. It's the same number you'd see in your agent builder, right next to the per dimension breakdown. We've tried to make the eval as transparent as possible.
Auto un-publish (along with confidence score analytics) is the piece that's still in the roadmap, not shipped yet. To your exact worry, yes when it does ship, the number and the reasoning behind it need to be visible to you before anything happens. We'd really not want a creator to be punished with a hidden reasoning behind the score!
For the "optimizing for the eval, not the actual answer" risk, yes that's honestly a glaring problem for any scoring system. We're fighting that by not just showing a blended score, but rather a breakdown across the different dimensions. So, "i'll try to hit 70 by any means", is kinda dealt by "don't have a outright failure in one dimension". Yes, to be fair, it's not a full fix to the risk, but that's where we have landed for now. And I'm really grateful that you brought up evals here, because that has been essentially our primary focus right now, to perfect quality over quantity.
@swapnanil_ray the dimensional breakdown instead of one blended number is a genuinely good call, it changes the incentive from "hit 70 by any path" to "don't tank any single dimension," which is a real constraint even if it's not a complete fix. fixed and visible next to the per-dimension score is exactly the transparency I was hoping to hear. appreciate you walking through the whole thing, not just the one-line answer.
@noctis06 Glad the per-message pricing resonates with you! Also on the context question, yes, it does.
Within a single conversation, the agent has the full thread: we send up to the last 30 messages of that conversation as context on every turn, so a follow-up 3 messages later is well within that window.
On top of that, we also run a long-term memory layer that persists relevant details across conversations, scoped to the user, so even if someone comes back days later in a new thread, the agent isn't starting from zero.
@zrk222 So each agent on Kopai is treated as an individual product. Therefore, all the agents listed on Kopai's marketplace has it;s own SEO that the creators can tune and optimise to market their product.
We are also trying a creator led economy, where we are collaborating with creators for them to make agents and reach out to their userbase for visibility. As our product actively let's you earn from your agent, that is an added incentive to share and market our product.
The eval gate that @galdayan pulled out of you is the right idea aimed at the wrong moment. It fires on edit, and the failure mode here is that there are no edits. An expert uploads once, the agent earns, and the pitch is explicitly that they step out of the loop. Meanwhile the expertise decays. Eighteen months on, the agent is confidently selling a 2026 answer and the expert has no signal it went stale, because the money is still arriving. Revenue is the one number that will never tell you your knowledge expired. So what re-triggers the eval when nothing has changed on the expert's side? A decay on the score, a re-cert prompt, something tied to the world moving rather than the file moving?
@galdayan @rabnoor_s Yes, this decay in the agent is actually a great catch. One thing on our pipeline is the auto-unpublish on feedback that has not been addressed, but the idea of a decay in score actually makes sense.
We were thinking of a yearly review system but this is something that we would want to address for our next launch
@suryansh_tiwari2 So currently we have implemented this feature called Chat Orientation. It is basically where the user is shown the goals and actions taken by the LLM. If they LLM drifts the user can manually update to realign the LLM. Another feature is we have a feedback loop, where users can leave feedback for the creator of the product. We believe it is important to keep the human or in this case the creator in the loop so as they can make sure that they are the ones who are driving the final direction.
What actually is on the roadmap right now: a response confidence score that pops up periodically on responses, feeding into an analytics layer that flags when an agent is underperforming and notifies the creator regularly. If in a case the score drops too low, the agent will get auto unpublished from the marketplace, until the creator fixes it. Would love your thoughts on this as well.
@petrkovacik This is probably still our hardest challenge. However, we have tried to tap into our founder as well as friends , family and colleagues to onboard our intial experts and we are trying to drive visibility through them.
We have reached out to few financial agencies, consultancies and so on as well.
Any help in this section would be highly appreciated



Kopai
Hey Omri, thanks for actually building something and being specific about what worked, that's more useful to us than a star rating alone.
On the eval gate, I appreciate you noticing that, it's been our main focus. Every agent gets scored across dimensions (system prompt quality, scope adherence, safety, plus KB/tool accuracy where relevant) before it can publish, and any edit re-triggers the check, so it's not publish-once-and-hope. The passing threshold is 70/100 and it's visible in the builder next to the per-dimension breakdown, not a black box.
About live vs frozen snapshot, it pulls the current published version, not a snapshot frozen at embed time. So when you update the agent, anyone who's embedded it sees the update too. Fair callout that this wasn't clear up front, we'll get it stated explicitly wherever embed is set up.
On technical categories being thin; Fair, and it tracks who's found us first, mostly consulting, coaching, and finance backgrounds. Getting devops/infra/security folks publishing is next, if you know anyone in that world sitting on knowledge worth packaging, send them our way!