What's your monthly LLM bill? How Much Are You Really Spending.

Question

We’re a data company that recently interviewed over 60 large enterprise VPs and 70+ startup engineers and CTOs. Here’s what’s intriguing: Despite the massive hype around Large Language Models (LLMs), some of the largest tech enterprises (5k+ employees) report LLM bills of less than $1000 per month. Meanwhile, some startups are burning through over $100k on LLMs like OpenAI, Claude, or even self-hosting models like LLaMA.

Is this a case of enterprises being cautious, or are they simply seeing through the LLM hype that startups are buying into?

Are you spending big on LLMs, or are you getting by with cheaper alternatives like LLaMA-CPP and other quantized models that run on CPUs? As a prelaunch team, we started with running llama cpp on a macbook to $500-$1k monthly openai bills over the past 6 months.
How much are you spending and what are your cost saving tricks!?

zane · Accepted Answer

Now the server is deployed, and now almost the bill is zero

Answer

I’ve  been curious about the costs too. What’s the best way  to  budget for LLM services and avoid surprises?

Answer

I was shocked by how high my bill was last month. I’m now focusing on optimizing my usage and exploring alternative providers to keep the costs under control. It’s definitely worth keeping a close eye on these expenses!

Answer

Using Anthropic's Claude API, my monthly LLM bill is around $100-200 depending on usage. It's not free, but the capabilities are impressive for certain use cases like content refinement, analysis, and Q&A. Definitely seeing ROI but keeping an eye on costs as usage grows!

Daniel James Harris · Answer

My LLM bill used to be crazy high, like hundreds per month. But then I optimized my prompts, fine-tuned my model, and moved to a cheaper provider. Now it's way more manageable, under $50/mo usually. Def took some trial and error to get the cost down while keeping the performance I needed tho.

Answer

I haven’t really tracked my monthly LLM bill closely. Do you have any tips on how to keep an eye on those costs?

Hassan Sajjad · Answer

We started small too. It's crazy how fast LLM costs can add up, but we’re trying to stay lean and mean.

Steven Shen · Answer

Some of our team's internal applications, such as data analysis and automation programs, consume about $200 per month on openAI. 
The new project we are working on is expected to consume $200-300 per day.

Answer

I keep our costs low by using quantized models, but it’s still a challenge balancing performance and budget.

Lihong Hicken · Answer

the cost of AI is high for us but what is more, is the annoying rate limit and we are slowing increasing rate limit. Not ideal.

YC J · Answer

@thommas_jack I feel this is similar to budgeting for compute. Running a sample job always provides us with great estimate on the full load. Cloudwatch’s native logging wasn’t helpful so we turned on our own logs

Answer

Startups spend more due to experimentation and scaling.

Answer

I’ve found that understanding exactly what features and usage are driving up my costs helps a lot. Sometimes, it’s just a matter of tweaking the configuration or switching to a different plan to save money.

Answer

It's interesting to think about the expenses for LLMs.

What's your monthly LLM bill? How Much Are You Really Spending.

Replies