Local inference profile
- Architecture
- Dense
- Parameters read per token
- 2B
- Maximum context
- 131,072 tokens
- FP16 KV cache per 1K tokens
- 0.015 GB
Published quantisations
| Format | Effective B/param | Estimated weights |
|---|---|---|
| BF16/FP16 | 2.00 | 10.0 GB |
| Q8_0 | 1.08 | 5.4 GB |
| Q6_K | 0.84 | 4.2 GB |
| Q5_K_M | 0.70 | 3.5 GB |
| Q4_K_M | 0.57 | 2.8 GB |
| Q3_K_M | 0.46 | 2.3 GB |
| Q2_K | 0.34 | 1.7 GB |