AI & data

How AI Workloads Change Your Cloud Bill

A glass network representing connected computing resources
AI-generated illustration.
Inside this article
  1. GPU machines cost more than GPU time
  2. Idle capacity and bursts shape compute costs
  3. Data movement adds network charges
  4. Storage costs include reads and retrievals
  5. Request counts do not predict the full bill
  6. Budget for AI cloud capacity and useful work

Adding AI to cloud services raises costs for GPU-backed machines, model and data storage, transfers between services, and spare capacity for traffic spikes. Managed AI services shift some of these costs to usage charges, but the total still depends on the work each request triggers.

Budget for the cost of delivering an acceptable result at the speed you need. GPU-hour rates and request counts alone do not show that cost. Model size, input and output length, traffic patterns, data location, and scaling settings all affect the bill.

GPU machines cost more than GPU time

A GPU, or graphics processing unit, handles the parallel calculations used by many AI models. Cloud providers generally sell it as part of a virtual machine that also includes processor cores, memory, network access, and sometimes attached storage. Those bundled resources affect the price even when the GPU is the main reason for choosing the machine.

For example, AWS lists a G6e.xlarge configuration with one L40S GPU, four virtual CPUs, and 32 GiB of machine memory. A workload that needs more GPU memory may require a larger machine, which also brings more of the other bundled resources. Comparing GPU-hour prices alone can therefore miss a substantial part of the compute bill.

Compare costs by useful output. For a service that runs a model, measure the cost per successful response or generated token while meeting its response-time target. Throughput, or how much work the service completes over time, also matters. So do batching, which processes requests in groups, the model choice, and the length of inputs and outputs.

One 2026 study that measured model performance and cost found that an AWS g5.xlarge had the lowest cost per generated token among the configurations it tested with 1,000 concurrent requests. It was neither the cheapest machine by the hour nor the fastest in that test. The result illustrates why hourly price does not determine cost per output; it does not establish a general ranking for other models or workloads.

Idle capacity and bursts shape compute costs

Interactive AI services need to respond quickly, so operators often keep some GPU capacity ready even when few requests arrive. Those idle machines still cost money. Scaling down cuts that cost, but starting new capacity takes time and can delay responses or make the service unavailable when traffic returns.

Cloud products handle this trade-off differently. AWS documents scale-to-zero, which shuts down capacity when idle, for certain SageMaker endpoints. Starting capacity after an idle period takes several minutes, and requests during that time can fail. With SageMaker asynchronous inference, a mode that queues requests for later processing, requests that arrive when capacity is zero can wait until capacity starts. This fits work that can tolerate delays better than a live chat response. Azure Machine Learning also distinguishes between online endpoints, which serve requests, and batch endpoints, which process jobs; its endpoint comparison documents scale-to-zero for batch endpoints.

Choose an autoscaling policy that meets the service’s response-time target. A chatbot that must answer immediately needs a different capacity plan from an overnight document-processing job. Test quiet periods and traffic spikes, then compare the GPU cost with the resulting delays or failures.

Sharing machines across models or containers can improve utilization when those workloads fit together. Batching requests can also keep a GPU busier, although waiting to form a batch may increase response time. These techniques help only when the combined workload still meets its performance requirements.

A technician examining a server rack
AI-generated illustration.

Data movement adds network charges

AI services move data among users, model-serving machines, storage, databases, and other cloud services. The price depends on the path: a transfer within one zone may receive different treatment from a transfer between zones, between regions, or out to the public internet. “Inside the same cloud” does not mean every transfer is free.

Google Cloud’s published pricing for its virtual private cloud (VPC), the network for cloud resources, lists traffic between virtual machines in the same zone using internal IP addresses as free. It lists traffic between zones in the same region at $0.01 per GiB. Its published North America Premium Tier internet egress rate is $0.12 per GiB for the first 1,024 GiB after an initial 1 GiB free tier. Standard Tier has different terms, including a 200 GiB free tier and a listed rate of $0.085 per GiB for the next pricing range. These are Google Cloud examples; rates and conditions differ by provider, service, region, and transfer path.

AWS’s S3 pricing page describes different rules for that service. Internet data transfer into S3 is exempt from S3 data-transfer charges, and the first 100 GB per month of internet data transfer out is free when aggregated across AWS services and Regions, with China and GovCloud excluded. Other exceptions apply, including to transfers between S3 and AWS services in the same Region.

To estimate network costs, map where model files and input data sit, where inference runs, and where responses go. Then check the provider’s rates for each path, especially cross-zone and cross-region traffic. For a closer look at how cloud providers charge for data movement, see The Real Cost of Cloud Egress and Data Movement.

Storage costs include reads and retrievals

AI storage bills cover more than model files. An organization may store training data, checkpoints that allow training to resume, generated outputs, and logs. The cost depends on how much data stays stored, where it lives, and how often a workload reads or moves it.

Lower-priced storage tiers often charge extra when you retrieve data. Google Cloud Storage lists retrieval fees of $0.01 per GiB for Nearline, $0.02 for Coldline, and $0.05 for Archive. Applicable operation and network charges can apply as well. A class that saves money for rarely accessed archives may cost more when an AI job repeatedly reads those files.

AWS S3 also separates storage charges from costs such as requests, retrieval, data transfer, and replication. Some infrequent-access classes have minimum object-size or minimum-duration conditions. Check those terms before moving large collections of files into a cheaper class.

Estimate storage in three parts: what the workload keeps, how often it reads that data, and where it sends the data next. That approach captures costs that a simple “gigabytes stored” estimate misses.

Request counts do not predict the full bill

Two AI services that receive the same number of requests can have very different costs. One request may contain a short question and receive a short answer; another may include a long document and generate a detailed response. When you run the model on your own cloud machines, those differences affect how much work the GPU does and how many requests the service can handle. For managed services, charges depend on the provider’s pricing model and the work performed.

Measure representative traffic rather than relying on request volume alone. Include input length, output length, the number of requests running at the same time, and the response-time target. Separate interactive traffic from batch jobs when their latency needs differ. If a managed model service is involved, use its billing units and actual usage records alongside cloud infrastructure charges.

A useful estimate also accounts for the supporting services around inference. A request may retrieve reference data, write logs, store an output, or call another service. Those operations add to the cost even if they do not run on the GPU.

Ethernet cables in a network rack
AI-generated illustration.

Budget for AI cloud capacity and useful work

Several common budgeting mistakes come from estimating only one part of the service:

  • Treating peak capacity as a constant. Multiplying peak GPU needs by every hour of the month can overstate costs when demand falls. Assuming capacity disappears instantly can understate costs and ignore warm-up delays.
  • Counting requests without measuring their size. Request volume does not show how much input the model reads or how much output it generates.
  • Counting only the GPU. The billed machine also includes CPU, memory, network, and potentially storage resources.
  • Assuming every transfer is free inside one cloud. Provider rules vary by service and path, including between zones and regions.
  • Choosing storage by its monthly rate alone. Frequent retrieval can add charges that outweigh savings on stored data.
  • Comparing hourly machine prices instead of results. A lower-cost machine per hour may deliver less output in the time available.
  • Committing before demand is clear. A discount does not help if the workload uses too little of the capacity or cannot take advantage of the offer.

Base the first budget on measured or tested workload behavior. Estimate the expected traffic range, then test how the service performs at quiet, typical, and peak demand. Track both total cloud spend and a workload-specific measure such as cost per successful response. Keep the latency and availability targets beside that measure so a cheaper configuration does not look better by ignoring its service impact.

Discounted capacity also carries conditions. AWS Spot Instances use spare capacity that AWS can reclaim, so they fit work that can tolerate interruption better than services that must stay available. Reserved capacity can reduce rates under eligible offers, but a long commitment made before demand stabilizes creates a risk of paying for unused capacity. Compare the discount with expected utilization and the workload’s ability to pause or move.

About this article

Created with AI assistance using published sources. Illustrations are AI-generated. Redbag does not claim hands-on testing or exclusive reporting. Check the linked sources for changing product details.

Our approach ↗

Keep connecting the dots.