A team spins up a few GPU instances to try out a new model. The pilot goes well. Everyone’s pleased, and the instances stay on over the weekend because nobody wrote down who owned them. When finance finally asks about the bill, the experiment has cost more than the project it was supposed to prove.
We made that story up, but you probably know the shape of it. AI workloads get expensive in ways ordinary applications don’t, and the old cost habits only half work. So here’s where the money really goes, and what a sensible team can do about it. Some people call this FinOps. Strip away the jargon and it’s finance and engineering looking at the same bill, on a schedule, with someone responsible for acting on it.
Why AI bills surprise people
Normal apps follow traffic. Quiet week, quiet bill. AI doesn’t behave like that. Training a model can swallow a huge block of GPU time over three days and then stop. Serving it to users (inference, in the jargon) keeps machines busy around the clock, asked or not.
And GPUs cost a lot more than regular servers. A forgotten general-purpose box costs you pennies, relatively speaking. A forgotten GPU box costs real money by Tuesday. Then there’s the storage for training data, the fees for hauling it between regions, and the three or four environments most teams end up running. Dev, test, staging, prod. Before long the total is well past whatever got budgeted.
The usual leaks
Most AI cloud bills leak in the same four places.
Idle machines, first. Notebooks and test clusters from last month’s experiment, still humming, because switching them off was nobody’s job. Too much hardware, second. Picking the biggest GPU “just to be safe” feels responsible. Plenty of small models run happily on something far cheaper. Data is the quiet one. Copies of big datasets in several regions, every version of every experiment kept forever, transfer charges nobody’s watching. They hide until they don’t. And the experiment with no finish line. A run meant to take a week, still going a month later, because nobody set a limit. Notice that none of these takes clever engineering. Somebody just has to look.
Get the tags on first
Here’s a quick test. Who owns the biggest GPU line on your last bill? If finding out takes more than a minute, you’ve met problem number one. The fix is tagging, and yes, it’s every bit as exciting as it sounds. Each resource gets a team, a project and an environment. (Google Cloud calls them labels.) Skip it and you’re left with one giant number, and every cost meeting turns into a guessing game. Once it’s done, the billing dashboards you already have (Cost Explorer on AWS, Cost Management on Azure, the billing reports on Google Cloud) will split the spend by project. No new software needed.
After that, put together a one-pager. Training spend on the left, inference in the middle, everything else on the right. Send it to the engineers each week, not only to finance, because people change what they run pretty quickly once they’ve seen the price tag.
Fit the hardware to the work
Picture a team serving a small classification model from a big multi-GPU machine, just because that’s what the training run used. Nobody went back and asked whether serving needs the same box. It usually doesn’t. Big GPU instances can run to tens of dollars an hour, so a few weeks of utilization numbers are well worth the trouble of pulling. If the card is mostly bored, go down a size. Or let a few models take turns on it.
Spot capacity is really a training thing. (AWS calls them Spot Instances, Azure has Spot VMs, and Google Cloud has Spot VMs plus the older preemptible ones.) The provider sells you spare machines at a discount, then keeps the right to grab them back with very little warning. That’s okay for a job that saves checkpoints, because a yanked machine only costs you whatever ran since the last save. It’s not okay for a customer-facing service. Keep that one off spot.
Inference that runs flat out all day is another matter. You’re paying for that capacity anyway, so savings plans or reserved instances are worth a look. Terms differ a lot from one provider to the next, so read your own contract before you believe anyone’s rule of thumb. Ours included.
Trim around the edges
Think about a dev box at 3 a.m. on a Sunday. Nobody’s touching it, and it’s still on the meter. Put a schedule on non-production environments so they power down overnight and at weekends. It’s about ten minutes of setup and among the cheapest wins going.
Then look at how the model gets served. You can batch requests, cache the answers that keep coming up, or swap in a smaller, compressed version of the model. Any of those can shave the compute per request. Check quality first, mind you. A cheaper model that answers worse hasn’t saved you anything. You’ve just swapped one problem for another.
Data deserves a pass as well. Old datasets and experiment outputs can drop to colder storage, and the files nobody’s going to open again can be deleted outright. Also, check how far the data travels before it gets processed. Cross-region transfer fees are sneaky.
Give every project an owner
Nobody fixes a problem that belongs to nobody. Pick one person for each AI project, give them a budget, and set an alert around eighty percent of the monthly number. That way they hear about it with time left to do something.
Then add a short monthly review. Read down the biggest line items. What jumped? Why? Does it need to be running at all? Twenty minutes is about right if the numbers are already on one page. A few months in, you’ll probably catch engineers raising cost in stand-ups without being asked, the way they do with latency.
Where cost fits in a cloud move
Plenty of teams leave cost control until the end, as a tidy-up job, and by then most of the damage is done. What helps is asking the cost question at every step, even when it feels early. Start with assessment: how much compute does this thing truly need? A lazy guess now becomes next year’s overspend. At migration time, choose instance types and storage tiers while the price list is open in another tab. And after go-live the job isn’t finished. Usage shifts, models grow, and last quarter’s sizing might be wrong by now.
Our AI cloud services are built around that idea: assess, migrate and optimize as one continuing effort, not three separate jobs. If you’re already on the other side of a migration, managing enterprise AI cloud services after migration covers the day-to-day work, and planning compute for training and inference goes deeper on the hardware side.
Six things to do this week
Tag every AI resource. Switch off anything idle. Check GPU usage and downsize where it’s low. Put a schedule on non-production environments. Name an owner and a budget per project. Book the monthly review.
No new tools. No big program. Do those six and you’ll very likely find money you didn’t know was leaving.
Modernize with confidence. Build for what's next.
Talk to our team about embedding AI into the core of your business.