Inferock-bench is a local proxy that sits between your app and OpenAI, Anthropic, Gemini, or OpenRouter shaped calls. It captures per-call token usage, failures, and retries, then generates an independent receipt showing what you were billed and how much you're actually overpaying for.
What's great
Ran npx inferock-bench against a day of normal traffic and it surfaced two retries I genuinely didn't know were happening - both silent, both billed. Pointing an existing SDK at it is really just swapping baseURL and apiKey, so the setup cost is close to zero for what it catches.
What needs improvement
The number it reports is only as good as its own copy of the provider's pricing table, and I couldn't tell from the CLI output how fresh that table was. A visible "prices last checked/updated" timestamp would make me trust the receipt without having to go verify it against the provider's pricing page myself.
vs Alternatives
Before this I was just eyeballing the provider dashboard totals month over month, which tells you the number changed but nothing about why. There isn't really a direct alternative that gives you the per-call receipt instead of the aggregate.
Ratings
Ease of use
Reliability
Value for money
Customization
Report
11 views
The failed-calls and retries breakdown is the part I'd use first. One case I keep hitting might not show up there: a tool call that returns 200 with a silently corrupted value. I measured this on Anthropic models, 40 calls, none flagged the value was wrong, so it bills as a clean success and the retry logic never fires. Can the receipt catch a call that looked fine but wasn't? Or is that out of scope by design?
Inferock Bench
@himashwetha_gowda Thanks, that's a clear answer and the right boundary to hold. The corrupted-200 case really does belong to a different layer. It needs a schema or an oracle on the payload and only the caller knows what the value should be. What you're proving from the outside is exactly the part I can't check myself today, so I'd keep it that sharp. Following the launch.
What I really want to know is how this handles historical data. Can I feed it a month of past logs and get a retroactive receipt or is it strictly forward-looking from install? I ask because the overspending I'm most curious about already happened and I'd love a way to audit it after the fact.
Inferock Bench
@kimberly_west really good question. Today it works by sitting in front of your live traffic, so the receipts start from the moment you point your SDK at it, it can't vouch for calls it never saw.
This is relevant to a problem I've had for months. I run agents that call out to multiple models depending on the task complexity and every so often the bill jumps in a way I can't explain from usage alone. If this can pinpoint whether that's failed calls, redundant retries or just legitimate scaling, I'd finally have an answer instead of a guess.
Inferock Bench
@yolanda_c_schneider Multi model agent setups are where this gets most interesting. Every call gets its own record whichever provider it went to, so a jump breaks down into failed calls, retries, etc. Would love to hear what you find.
Inferock Bench
Hey PH
We built inferock-bench because we kept paying for AI answers that died mid sentence, and nobody could tell us where the money went.
Providers give you totals. They don't give you the per call receipt you'd need to prove which answer broke, which retry ran, or which token count changed. The company that charges you also decides what counts as a failure and keeps the only detailed records.
inferock-bench runs locally as a proxy in front of OpenAI, Anthropic, Gemini, OpenRouter shaped calls. Point your existing SDK at it (change two settings: apiKey and baseURL), and it captures every call as an independent, per call record. Your provider key never touches our servers, it's used locally only, attached to provider requests.
What it catches:
- Answers cut off mid stream that still got billed
- Empty replies with billed tokens attached
- Token counts that don't match visible output
- Retries that may have silently doubled a charge
- Cache discounts you may be missing on your invoice
Every run reports a receipt: spend observed, bill-bounded money loss, time loss, and a separate "invoice-check exposure" line that never gets summed into money loss, because we don't want a louder headline at the cost of a weaker claim.
Run it in about a minute: npx inferock-bench
Its open source (FSL-1.1-Apache-2.0, converts to Apache-2.0 in 2 years).
Question for this community: has anyone here actually disputed an AI provider bill and gotten a credit? What worked?
@bharath_koneti @tehreem_fatima5 Appreciate you actually answering the question, that matches what we kept hearing from others too. Without per request records the conversation with a provider ends pretty fast. You walk in with evidence instead of a feeling.
@tehreem_fatima5 Thank you!
Genuinely hoping it saves a few of those support conversations.
DROP
Retries are the part I’d put in giant font. ‘$0.04/call’ means nothing if a successful job secretly takes 6 calls. I want cost per accepted outcome.
Inferock Bench
The bill dispute question is interesting. have you personally managed to get a provider to credit a charge after showing them one of these per call receipts or is that still something you are testing?
Inferock Bench
@jeremy_loomis honest answer: no credit to brag about yet, that's exactly why we threw the question to the community. What we can stand behind today is the receipt itself, knowing which call failed and what it cost, instead of arguing from a monthly total.
Does anyone know of any tools that can work with web-based logins (claud.ai etc)
Inferock Bench
@jay_janarthanan1 we don't touch that layer. inferock-bench works where there's an API key and a baseURL to point somewhere, and web app subscriptions don't expose the per call detail we'd need. If someone has cracked that curious to see it too.