Cloud GPU Pricing

Common questions

What is the cheapest cloud GPU?

The cheapest listing we currently track is the Nvidia GTX 1050 Ti at Vast.ai, at $0.02 per GPU per hour. The cheapest option for a given workload is usually a different one: the smallest GPU whose VRAM fits your model, in a region you can actually get capacity in.

How often are these GPU prices updated?

We track 3,177 GPU configurations from 75 providers. Listings from providers we scrape are re-checked as often as every 15 minutes; the rest are reviewed manually. The stamp at the top of this page shows when we last refreshed a listing, which is not the same as a price having moved.

What does $ / GPU / hr mean?

Every price on this page is normalized to one GPU for one hour, in US dollars, so instances of different sizes can be compared directly. An 8xH100 instance at $24.00/hr is listed as $3.00 per GPU per hour. Prices in other currencies are converted at current ECB rates.

What is the difference between on-demand, spot, and reserved GPU pricing?

On-demand is the pay-as-you-go rate, billed by the second or hour with no commitment.

Spot (or preemptible, or interruptible) uses spare capacity at a discount, and the provider can reclaim the instance with little notice. Suited to checkpointed training and batch jobs, not to serving traffic.

Reserved trades a commitment of one month to several years for a lower hourly rate. Larger clusters are usually quoted as a custom contract rather than a list price.

Which GPU is best for AI training vs inference?

It depends on the workload, but the specs that decide it are the same:

• Memory capacity and bandwidth: more VRAM allows larger models and batch sizes
• Tensor cores: specialized units for the matrix math AI models run on
• FLOPS: raw compute throughput
• Interconnect: SXM and NVLink move data between GPUs faster than PCIe, which matters once a job spans several GPUs

For training, VRAM and interconnect dominate: H100, H200, B200 and A100 are the usual choices. For inference, a smaller GPU whose VRAM fits the model is often far more cost-effective, such as an L40S, A10 or RTX 4090.

For a deeper guide, see Tim Dettmers' GPU recommendations.

How much VRAM do I need?

As a rough rule for inference, a model needs about 2 GB of VRAM per billion parameters in FP16, or about 0.5 GB per billion in 4-bit quantized form, plus headroom for the KV cache and runtime overhead, which grows with context length. Training needs several times more, since optimizer states and gradients are held alongside the weights.

Multi-GPU instances add total memory, but that memory is not automatically pooled: the model has to be sharded across GPUs, and the interconnect then becomes the limit.

Nvidia vs AMD: which is better for AI?

Nvidia's advantage is CUDA. It is the mature, default software stack, and nearly every framework and kernel targets it first.

AMD's advantage is often price and raw hardware specs, particularly memory capacity on the MI300X and MI325X. Its ROCm stack has improved considerably but still lags in coverage of tooling and less common operations.

In practice, teams with the engineering capacity to work around gaps can run AMD economically. Everyone else pays a premium for CUDA and saves the time.

What is the difference between CUDA cores and tensor cores?

Both are processing units on Nvidia GPUs, with different jobs. CUDA cores handle general-purpose parallel math, such as rendering and simulation. Tensor cores are specialized for the matrix multiply-accumulate operations that dominate neural network training and inference, at reduced precision.

The closest AMD equivalents are stream processors and matrix cores, though they are not comparable one to one.

Can I run AI workloads on a CPU instead?

Yes. PyTorch and other frameworks run on CPUs, and Apple Silicon in particular does reasonably well on small models thanks to unified memory and its neural engine.

The gap is throughput. A GPU applies the same operation across thousands of cores at once, with far more memory bandwidth, so it is typically an order of magnitude or more faster on the matrix math these models are made of. CPUs remain viable for small models, low request volumes, and experimentation.