AMDData Center GPU
AMD Instinct MI300X
Based on CDNA 3 architecture. 192GB memory with 750W TDP.
FP16 Performance
1307
TFLOPS
Memory Capacity
192
GB
Power Draw
750
Watts TDP
Performance per Dollar
0.05 TFLOPS / $
Based on list price
Subject to market fluctuation
Benchmark Results
Published results grouped by model variant and scenario definition
7 resultsOpen Category View
Mistral 7B Instruct v0.3
FP8 Dense Serve · FP8
text-generation · vLLM · Batch 16 · Seq 4096 · 4096 in / 256 out
throughput
5,450 tokens/s
Latency P50
64 ms
Throughput
5,450 tokens/s
Power
730 W
Dense Mistral serving profile on AMD MI300X.
Gemma 2 9B Instruct
INT4 Chat Serve · BF16 · INT4
text-generation · TensorRT-LLM · Batch 4 · Seq 2048 · 2048 in / 128 out
latency
50 ms
Latency P50
50 ms
Throughput
940 ms
Power
700 W
Gemma latency profile on AMD MI300X.
Llama 3.1 8B Instruct
INT4 Serve · FP8 · INT4
text-generation · TensorRT-LLM · Batch 32 · Seq 2048 · 2048 in / 256 out
throughput
7,600 tokens/s
Latency P50
78 ms
Throughput
7,600 tokens/s
Power
720 W
Dense Llama INT4 serving profile on AMD MI300X.
Qwen2.5 7B Instruct
FP8 Low-Latency · FP8
text-generation · vLLM · Batch 1 · Seq 4096 · 4096 in / 256 out
latency
58 ms
Latency P50
58 ms
Throughput
760 ms
Power
710 W
Low-latency Qwen profile on AMD MI300X.
Llama 3.1 8B Instruct
FP8 Serve · FP8 · FP8
text-generation · TensorRT-LLM · Batch 1 · Seq 1024 · 1024 in / 128 out
latency
46 ms
Latency P50
46 ms
Throughput
—
Power
700 W
Demo single-request latency row for UX validation.
Qwen2.5 7B Instruct
BF16 Serve · BF16
text-generation · vLLM · Batch 16 · Seq 2048 · 2048 in / 256 out
throughput
6,100 tokens/s
Latency P50
66 ms
Throughput
6,100 tokens/s
Power
735 W
Demo multi-tenant throughput row for UX validation.
Llama 3.1 8B Instruct
FP8 Serve · FP8 · FP8
text-generation · TensorRT-LLM · Batch 8 · Seq 4096 · 4096 in / 512 out
throughput
5,400 tokens/s
Latency P50
58 ms
Throughput
5,400 tokens/s
Power
720 W
Demo composite throughput row used to validate public benchmark cards.
Specifications
ManufacturerAMD
CategoryData Center GPU
ArchitectureCDNA 3
Process Node5nm + 6nm
Form FactorOAM
CoolingAir
VRAM192 GB
VRAM Type—
Interconnect Bandwidth896 GB/s
Tensor / Matrix Cores304
Supported PrecisionsFP16, BF16, INT8
TDP750 W
FP16 (Dense)1307 TFLOPS
FP32 (Dense)163 TFLOPS
Release Date2023-12-01
Price (USD)$25,000