Local inference profile
- Architecture
- Dense
- Parameters read per token
- 32B
- Maximum context
- 131,072 tokens
- FP16 KV cache per 1K tokens
- 0.262 GB
Published quantisations
| Format | Effective B/param | Estimated weights |
|---|---|---|
| BF16/FP16 | 2.00 | 64.0 GB |
| Q8_0 | 1.08 | 34.6 GB |
| Q6_K | 0.84 | 26.9 GB |
| Q5_K_M | 0.70 | 22.4 GB |
| Q4_K_M | 0.57 | 18.2 GB |
| Q3_K_M | 0.46 | 14.7 GB |
| Q2_K | 0.34 | 10.9 GB |