Mistral Small 3.2 24B

mistralai/Mistral-Small-3.2-24B-Instruct-2506

24BDenseText generationImage → textText inputImage input

Mistral AI · 1 recorded configuration

Measured setup

Hardware

2× RTX 3090 · GPU-resident

Runtime and tool

llama-bench

Quantisation

i1-Q5_K_M

MTP draft depth

MTP off

Every option opens a recorded setup. Where the corpus recorded nothing at this page's other settings, the option names the fields it switches with it. Allocation and runtime detail stay in the technical record below.

PP · left axisTG · right axis

View exact measurements

Exact context depth and measured fresh prefill, bulk prefix, and decode rates for Mistral Small 3.2 24B
Context depth (tokens)	Fresh prefill PP (tok/s)	TG (tok/s)
0	2,284.10	72.70
2,048	2,216.60	70.87
4,096	2,153.00	68.82
8,192	2,038.30	65.44
16,384	1,855.10	59.94
32,768	1,553.60	51.02

Concurrent-stream measurements

Concurrent-stream throughput at the recorded existing-prefix depth for Mistral Small 3.2 24B
Streams	Aggregate PP (tok/s)	Aggregate TG (tok/s)	TG / stream (tok/s)	Aggregate all (tok/s)
1	2,284.10	72.70	—	—
1	2,170.40	72.29	72.29	203.31
2	2,198.70	119.82	59.91	324.14
4	2,232.60	165.58	41.39	432.57

Technical record and provenance

Run details

Source run ID: rtx3090x2-mistral-small-3.2-24b-iq5km-2026-08
Hardware: 2× RTX 3090
Runtime: llamaCpp · llama-bench fc6545d (build 50)
Weight placement: gpu-resident
Split ratio: 1:1
KV-cache precision: Q8_0
Flash Attention: Enabled
Batch size: 2,048
Microbatch size: 512
Prompt tokens: 512
Generated tokens: 256
Speculative decoding: None (standard decoding)
Measured repetitions: 3
Notes: i1-Q5_K_M, batch 2048/512, all layers on GPU, tensor split 1:1. Concurrency rows from llama-batched-bench (aggregate rates). Model is not in the calculator catalog, so this record is archival until it is added.
Source article: /local-ai/rtx-3090-for-local-ai-inference/

Compare measured entries