Cloud GPU pricing

The lowest listing we currently track is the Nvidia GTX 1050 Ti at Salad, $0.020 per GPU per hour on-demand. For the cheapest GPU by VRAM requirement, class and current availability, see our cheapest cloud GPUs guide.

The cheapest column is the lowest on-demand listing for that GPU: often a marketplace host, a small configuration or a promotional rate. The median is what a typical provider charges on demand: each provider's own median first, then the median across providers, so a provider listing thirty instance sizes counts once. Budget on the median, then open the GPU's page to see who sells below it and on what terms.

We track 5,482 GPU configurations from 85 providers. Listings from providers we scrape are re-checked as often as every 15 minutes; the rest are reviewed manually. The Providers figure at the top of this page says when we last refreshed a listing, which is not the same as a price having moved.

Every price on this page is normalized to one GPU for one hour, in US dollars, so instances of different sizes can be compared directly. An 8xH100 instance at $24.00 /hr is listed as $3.00 per GPU per hour. Prices in other currencies are converted at current ECB rates.

On-demand is the pay-as-you-go rate, billed by the second or hour with no commitment.

Spot (or preemptible, or interruptible) uses spare capacity at a discount, and the provider can reclaim the instance with little notice. Suited to checkpointed training and batch jobs, not to serving traffic.

Reserved trades a commitment of one month to several years for a lower hourly rate. Larger clusters are usually quoted as a custom contract rather than a list price.

It depends on the workload, but the specs that decide it are the same:

• Memory capacity and bandwidth: more VRAM allows larger models and batch sizes
• Tensor cores: specialized units for the matrix math AI models run on
• FLOPS: raw compute throughput
• Interconnect: SXM and NVLink move data between GPUs faster than PCIe, which matters once a job spans several GPUs

For a deeper guide, see Tim Dettmers' GPU recommendations.

For training, VRAM and interconnect dominate. H100 and H200 are the usual choices, B200 and B300 the current Blackwell generation, and GB200 and GB300 the rack-scale systems above them. The A100 is still widely listed and costs less per hour.

For inference, a smaller GPU whose VRAM fits the model is often far more cost-effective: an RTX PRO 6000 Blackwell at 96 GB, an L40S at 48 GB, or an RTX 5090 or L4 for smaller models.

As a rule of thumb for inference, add three terms: the weights, the KV cache and runtime overhead.

• Weights: about 2 GB per billion parameters at 16-bit, 1 GB at 8-bit and 0.55 GB in 4-bit quantized form
• KV cache: set by the model's layer count, KV heads and head size, and it grows with context length and with concurrent requests. Across open-weight models the rate runs from about 35 KB per token to over 300 KB, so a 32K context costs roughly 1 to 10 GB depending on the architecture
• Runtime overhead: about 2 GB for the CUDA context, activations and sampling buffers

Then leave headroom. Our fit tables treat a model as fitting when it lands within 90% of the card's advertised VRAM, because vLLM and SGLang reserve the rest by default and a model sized at 95% of a card does not start.

Two caveats: training needs several times more than inference, since optimizer states and gradients are held alongside the weights. And multi-GPU instances add total memory, but that memory is not automatically pooled: the model has to be sharded across GPUs, and the interconnect then becomes the limit.

Nvidia's advantage is CUDA. It is the mature, default software stack, and nearly every framework and kernel targets it first.

AMD's advantage is often price and raw hardware specs, particularly memory capacity on the MI300X, MI325X and MI355X. Its ROCm stack has improved considerably but still lags in coverage of tooling and less common operations.

Both are processing units on Nvidia GPUs, with different jobs. CUDA cores handle general-purpose parallel math, such as rendering and simulation. Tensor cores are specialized for the matrix multiply-accumulate operations that dominate neural network training and inference, at reduced precision.

The closest AMD equivalents are stream processors and matrix cores, though they are not comparable one to one.

Yes. PyTorch and other frameworks run on CPUs, and Apple Silicon in particular does reasonably well on small models thanks to unified memory and its neural engine.

The gap is throughput. A GPU applies the same operation across thousands of cores at once, with far more memory bandwidth, so it is typically an order of magnitude or more faster on the matrix math these models are made of. CPUs remain viable for small models, low request volumes, and experimentation.