ngrok AI Gateway
One private gateway for every AI model
541 followers
One private gateway for every AI model
541 followers
ngrok AI Gateway provides one hosted gateway for every model: public providers, custom endpoints, and the models you run yourself. Use one key and one URL to route across OpenAI, Anthropic, and self-hosted models with observability, access control, and fallbacks built in. Your private models connect through ngrok’s network, so they sit beside hosted providers without being exposed to the public internet.
Payment Required
Launch Team / Built With
Render: Get up to $100K in startup creditsThe fastest way to launch and scale your startup in the cloud
Promoted
Hey Product Hunt 👋
I'm Niji, a product manager at ngrok. Today we're launching ngrok.ai, ngrok's AI Gateway.
For years, ngrok has helped developers connect their applications and services in minutes instead of days.
As I started building with AI, I ran into similar infrastructure problems at the application layer.
An application might start with OpenAI, then Claude for another use case. As newer, faster, or more affordable models became available, I would create more accounts and update my code just to try them. Eventually, more specialized needs would lead me to run fine-tuned or task-specific models on my laptop, private GPUs, or internal cloud infrastructure.
Before long, I was managing multiple gateways and SDKs, sharing provider keys across configuration files and vaults, checking usage in several dashboards, maintaining complicated fallback logic, and accidentally exposing models that were supposed to remain private.
If any of this sounds familiar, it is why we built ngrok.ai. It gives you one hosted gateway for managing models across providers, private infrastructure, and your own hardware.
One URL for every model
Getting started is simple. Point your SDK at https://gateway.ngrok.ai with your ngrok.ai access key, and begin routing requests to public providers, custom endpoints, and models you run yourself.
It works with all popular SDKs like OpenAI, Anthropic, and Vercel AI, so you can easily swap models and providers without rebuilding your entire application.
Aside from being a hosted AI Gateway, we enable you to:
Connect self-hosted models privately
Route to a model running on your laptop, local GPU, or private network without complex networking, opening inbound ports or dealing with IPs.
Build fallbacks into the gateway
Define a list of models and when a model or key fails, we will make another attempt or route the request to a healthy alternative.
Use credits to make requests
Leverage ngrok.ai to make requests against OpenAI, Anthropic, z.ai and more without having to create your own accounts with each provider.
Use your existing provider keys
Don't want to use our accounts? No worries, you can bring your own OpenAI, Anthropic, or custom provider keys that you already.
Control access by application or developer
Create separate access keys and decide which providers and models each one is allowed to call, and which keys each model should use, whether ours or yours.
See usage across your entire model stack
Track tokens, latency, errors, models, providers, and estimated cost in one place instead of piecing together several provider dashboards.
Manage everything through the dashboard or API
Set up gateways, keys, providers, access rules, and routing from your own tooling using our API or directly in the ngrok.ai dashboard.
Who we're building this for
ngrok.ai is for developers and platform teams that want the freedom to use the right model for each job without worrying about how to scale and maintain an ai gateway themselves and or taking on another infrastructure project every time their model strategy changes.
We're especially interested in hearing:
How are you routing between models today?
Are you running any models on your own infrastructure?
Which gateway features would make your AI stack easier to manage?
We'll be here throughout the launch to answer questions and hear what you think. Thanks for checking it out.
@ngrok is so back.
If you've ever needed to share a local app or demo something live, there's one thing that strikes about this product. Developers love it. I mean millions of developers - look at this wall of love - including people at @GitHub, @OpenAI, @Vercel, and much, much more.
@cassidoo put it simply: "ngrok saves me so much time during development. What a tool."
pumped to see them back [on Product Hunt]. S/O ?makers, already looking for your future launches.
@cassidoo @fmerian ngrok has been core in my dev setup for a very long time to test real world app scenarios during development. one gateway for every hosted model sounds interesting.
@gokuljd lfg! if you get a chance to give ngrok.ai a spin, make sure to add your review here: producthunt.com/products/ngrok-ai-gateway/reviews/new
@gokuljd Thanks so much for using ngrok over the long haul! If there's anything you'd like to see fixed or have us add (aside from this AI Gateway, of course!), I'd love to hear about it.
the fallback is the feature i would think hardest about, because for anything that produces text a fallback is a silent quality change.
if a code call falls back you find out, something breaks or the tests go red. if a customer facing reply falls back to a weaker model, nothing breaks. the reply still reads fine, it is just slightly worse, and you find out from a complaint two days later while looking at the wrong model in your logs. the gateway is the only component that knows which model actually answered, so i would want that on the response itself rather than only in a dashboard.
the other one is retries. a fallback triggered by a timeout is not the same as one triggered by an error, because on a timeout the first call may well have completed on the provider side. harmless for a plain completion, not harmless once a tool call is attached to it, which is most agent traffic now. does the gateway treat timeouts as retryable by default, and is there a way to mark a request as do not retry?
@jernej_jan_kocica This is really great feedback.
For your first point—I've always found that responses from the APIs of public providers like OpenAI and Anthropic contain some kind of `model` key that you could build logic around in your apps. It'll get flagged in the dashboard, just like you asked for, but you could build some error handling/logs around what happens when that fallback model is the one that responds. We'll dig into this one some more and see if we can add some headers at the gateway your app could also consume.
On your second question, timeouts only trigger retries if you've specified fallback models though the `model` key. If there isn't one, you just get an error, which is kind of akin to marking it as do not retry. This does feel like something we should look into implementing as a very clear attribute of any provider setup.
@joel_hans2 both of those are good answers, thanks for actually digging in.
one refinement on the do not retry attribute: i would put it on the request rather than on the provider setup. whether a call is safe to retry depends on what the call does, not on where it goes. the same model at the same provider will serve an idempotent summarise in one code path and a tool calling turn in another, and if the flag lives on the provider config you end up duplicating the whole provider just to express that difference. a header on the request costs nothing and lets the caller say it, and the caller is the only one who knows.
the model header on the response is the one i would ship first though. it is the difference between debugging what you configured and debugging what actually happened.
I like that you can start with your own provider keys and still get a single place to track usage, latency, and costs. Jumping between multiple dashboards gets old fast, so having that consolidated is a practical improvement.
@henry_habib Yep yep, you have complete control over whether you bring your existing providers keys (BYOK) or use ngrok-managed ones. Either way, it all ends up on the same request/usage logs.
We actually just added more detailed breakdowns of usage data we were already collecting, so now it's really easy to see things like the average cost per request or per-provider failure rate. This is me just playing around with a few requests this morning (and having forgotten to bring my local Ollama model online before sending the request):
How does ngrok AI Gateway help developers handle these issues compared to building their own layer? congrats team!
@hamza_afzal_butt
as with most software products, if you know how to write software and have the budget to host/distribute it, you can totally roll your own solution if your needs are simple.
similar to a traditional API gateway, if all you need to do is proxy requests, that's really easy to build your own solution. once you start getting into solving for things like rate limiting, observability & reporting, request manipulation, authentication, authorization, and role-based permissions, caching, building all these things in-house compared to picking an off the shelf solution is a decision that most teams have to decide if they'd rather pay the cost of a product or the cost of maintaining their own solution. we do the same thing internally with other tools. sometimes we write our own, other times we pay for a solution.
our goal with ngrok.ai is to provide one unified interface for any kind of LLM inference need and make it super easy to use while offering the depth to solve as many of the pain/friction points as possible. we hope that the benefits that people get from using it and the problems they don't have to think about solving themselves make it worth the cost to them. we're always on the lookout though for ways to keep improving it each week :)
I've had so many requests to build this exact solution in house and rejections keep coming back. Most teams can't afford to build/manage these things. Great job solving this problem!
@yazan_barakat
haha yep, it's a surprising amount of work for something that seems simple on the surface. we're hoping that users feel that using it is saving them a meaningful amount of time and/or avoided friction compared to going without it or building their own in-house solution :)
@yazan_barakat Oh, we'd love to chat with you about what those requests look like and why they haven't stuck. We're here if you're interested!
Hello Niji, congratulations on the launch. I like that you are making it easier to switch between different AI providers without rewriting everything. That sounds like it could save me a lot of effort as projcts grow.