What's your monthly LLM bill? How Much Are You Really Spending.
YC J
22 replies
We’re a data company that recently interviewed over 60 large enterprise VPs and 70+ startup engineers and CTOs. Here’s what’s intriguing: Despite the massive hype around Large Language Models (LLMs), some of the largest tech enterprises (5k+ employees) report LLM bills of less than $1000 per month. Meanwhile, some startups are burning through over $100k on LLMs like OpenAI, Claude, or even self-hosting models like LLaMA.
Is this a case of enterprises being cautious, or are they simply seeing through the LLM hype that startups are buying into?
Are you spending big on LLMs, or are you getting by with cheaper alternatives like LLaMA-CPP and other quantized models that run on CPUs? As a prelaunch team, we started with running llama cpp on a macbook to $500-$1k monthly openai bills over the past 6 months.
How much are you spending and what are your cost saving tricks!?
Replies
Steven Granata@steven_granata
consider alternatives like quantized models for cost savings.
Share
I am spending a lot, but my monthly bill is zero, because I use free version.
Now the server is deployed, and now almost the bill is zero
I’ve found that understanding exactly what features and usage are driving up my costs helps a lot. Sometimes, it’s just a matter of tweaking the configuration or switching to a different plan to save money.
I keep our costs low by using quantized models, but it’s still a challenge balancing performance and budget.
Spending over $100k on LLMs sounds wild. Is it really worth it? Would love to hear if anyone’s seen big returns.
I’m curious if enterprises are just holding back on LLMs or if they’ve found better ROI elsewhere.
@maximo_hartliness doubt that better ROI can be found as an Apple to Apple case to LLM investment. Cost cutting seems to be a main theme across enterprises tho
Theysaid
the cost of AI is high for us but what is more, is the annoying rate limit and we are slowing increasing rate limit. Not ideal.
Some of our team's internal applications, such as data analysis and automation programs, consume about $200 per month on openAI.
The new project we are working on is expected to consume $200-300 per day.
I haven’t really tracked my monthly LLM bill closely. Do you have any tips on how to keep an eye on those costs?
@claudia_lyons turning on token count logging on everything really helped us especially if there are automated function calls involved. We chained cheap models to shrink token size before sending things to a bigger model. And the cheapest solution is adding that magic line to everything :” provide answer within X words. “
I’ve been curious about the costs too. What’s the best way to budget for LLM services and avoid surprises?
@thommas_jack I feel this is similar to budgeting for compute. Running a sample job always provides us with great estimate on the full load. Cloudwatch’s native logging wasn’t helpful so we turned on our own logs
It's interesting to think about the expenses for LLMs.
Using Anthropic's Claude API, my monthly LLM bill is around $100-200 depending on usage. It's not free, but the capabilities are impressive for certain use cases like content refinement, analysis, and Q&A. Definitely seeing ROI but keeping an eye on costs as usage grows!
Bruh my AWS bills were crazy high from all the LLM API calls... like hundreds per month 😱 Switched everything to a dedicated A100 GPU in a Coreweave cloud instance. Costs way less now, only like $1-2/day. Takes some DevOps skills to set up but totally worth it to save that 💸💸💸
My LLM bill used to be crazy high, like hundreds per month. But then I optimized my prompts, fine-tuned my model, and moved to a cheaper provider. Now it's way more manageable, under $50/mo usually. Def took some trial and error to get the cost down while keeping the performance I needed tho.