AIFIRST Shoehorn: hardware-aware mixed-precision quantization that runs in the browser
The brief
Shoehorn is a locally run, browser-based tool that quantizes language models with hardware-aware mixed precision, assigning a precision level per tensor rather than applying a preset and targeting about 99.99% of available memory.
Key points
- It scans popular Hugging Face models to work out what fits a given machine and writes standard GGUF v3 files for llama.cpp.
- It supports macOS on Apple Silicon, Linux on x86-64, and Windows on x86-64.
- The quantizer core is written from scratch in Rust.
Sources
- HNnotactuallytreyanastasio.github.io