Home / Open Source / Inference
All Inference items under Open Source on AIFIRST News, newest first. 20 items.
A widely-shared r/LocalLLaMA post argues Ollama is the wrong default for running local models, pointing to performance limitations and better alternatives for serious local inference. The thread has sparked broad discussion about which tools actually matter for production local AI deployments.
A developer found the official llama.cpp build is not optimized for AMD Strix Halo (gfx1150) hardware and shared an alternative setup guide for achieving the highest possible throughput on that chipset. The post notes roughly 90% of Strix Halo users are still running the unoptimized stock build.
The ik_llama.cpp fork merged multi-token prediction support for Qwen3.8-Flash-Next via PR #2369, pushing throughput from roughly 45 to 90 tokens per second on compatible hardware. The change is now on main and requires no fork or patch. Users can use either an integrated head or a separate model file for MTP.
project-continuity is an open-source protocol for persisting AI coding agent state across sessions as plain files on disk. It makes individual sessions disposable while preserving project goals, decisions, and progress. The lightweight design works with any agent framework that can read and write text files.
WebLLM enables high-performance large language model inference directly in the browser using WebGPU, with no server required. Models run fully client-side, making inference private and zero-cost after initial download. The project supports a range of open-weight models and is available as an open-source library.
Vision support has been merged into DeepSeek-V4-Flash-Vision-Exp, with Unsloth GGUF quantized variants now available on Hugging Face. The update enables local users to run the vision-capable DeepSeek Flash model through llama.cpp-compatible tools. Multiple quantization levels are available for different hardware configurations.
A local inference enthusiast documented achieving 280 tokens per second on Qwen3.8 27B using two AMD Radeon RX 9700 GPUs with a 940k token KV cache. The writeup covers driver setup, ROCm configuration, and the specific llama.cpp flags that unlocked the result. Dual R9700 performance figures are scarce, making this a useful reference for the AMD open-source inference path.
In addition to the Hugging Face platform acquisition, Nvidia would effectively acquire the llama.cpp project and its core development team, which joined Hugging Face in February. llama.cpp is the dominant open-source runtime for running local LLMs on consumer hardware. This concentration raises questions about whether the project will remain hardware-neutral and freely licensed going forward.
llama.cpp has merged support for Qwen 3.8 Flash Next, the new efficient Qwen model variant. GGUF quantizations are now available, with one user reporting 55 tokens per second on a four-GPU RTX 3090 setup using Q4 quantization. The merge makes the model accessible through the standard llama.cpp toolchain without additional patching.
JetBrains has shipped a local AI coding assistant for its IDE lineup, selecting Qwen3.6 27B as the underlying model after evaluating both Qwen3.6 and Qwen3.8, finding the earlier version's reasoning characteristics better suited the coding assistant use case. The integration offers a local, privacy-preserving alternative to cloud AI assistants inside professional development environments. JetBrains is one of the largest IDE vendors to ship a bundled local model solution.
The ConvRot quantization method, which outperforms standard UDQ8_K_XL in accuracy benchmarks, has been integrated into the llama-cpp-turboquant fork. ConvRot applies a learned rotation to model weights before quantization, reducing the accuracy loss from low-bit formats. The fork is available on GitHub for users who want better quantization quality beyond standard GGUF methods.
A developer ran a 3-day benchmark of Inco AI's DFlash 2 speculative decoding PR in llama.cpp against plain decoding, MTP, n-gram lookup drafters, and the previous DFlash 1 on Qwen 3.8 27B. DFlash 2 introduces a trained drafter specifically for Qwen 3.8 27B and shows meaningful throughput improvements over all comparison methods.
A developer released a llama.cpp fork with HIP optimizations targeting AMD GFX906-class GPUs including the Mi50, Mi60, Radeon VII, and GCN HIP cards. These cards have minimal support in the mainline llama.cpp codebase, leaving many AMD users without an optimized inference path.
AI-Researcher is a self-hosted Python 3.11 dashboard that pulls AI news from roughly 60 sources including RSS, Reddit, Hacker News, arXiv, Hugging Face, GitHub, Google News, and X. It deduplicates and clusters items with optional nomic-embed-text embeddings and ranks them with engagement normalization that weights corroboration across sources. A local Ollama model summarizes the high-priority items. All processing is local with no mandatory third-party API keys. The repository had 4 commits and 0 stars when checked.
llama.cpp is a C/C++ implementation of large language model and vision language model inference, built on the ggml tensor library and MIT licensed. It ships a command line interface, an OpenAI-compatible API server, and a web chat UI, installable as an app, via Docker, from pre-built binaries, or from source. Backends cover Apple Silicon with ARM NEON, Accelerate and Metal, x86 with AVX through AVX512 and AMX, NVIDIA CUDA, AMD HIP, Vulkan and SYCL, plus hybrid CPU and GPU inference. The repository showed 124.9k stars and 22.0k forks with 778 open issues and roughly 1.4k open pull requests, and quantization spans 1.5-bit to 8-bit so output quality depends on the level chosen.
Huzzah is an experimental editor that uses pseudocode files (.hz) instead of natural language prompts to direct AI code generation. When a .hz file is saved, Huzzah generates real code; subsequent edits capture only the diff, reducing token consumption and preserving a persistent record of developer intent. The approach positions coding as design, with specs that are language-agnostic and reusable across multiple implementation targets. The author notes limitations including scalability concerns and challenges with cross-file dependencies.
Vomit is a small open-source tool that intercepts Claude agent mode's intermediate reasoning output and pipes it through a local LLM to produce more readable English summaries. It runs fully locally with no telemetry, supports Llama.app and Ollama backends, and can be used either as an integrated hook replacing Claude's displayed output or as a standalone session monitor via the 'vomit tail' command. The tool targets the verbosity of Claude's token output during agentic tasks.
Shoehorn is a locally run, browser-based tool that quantizes language models with hardware-aware mixed precision, assigning a precision level per tensor rather than applying a preset and targeting about 99.99% of available memory. It scans popular Hugging Face models to work out what fits a given machine and writes standard GGUF v3 files for llama.cpp. It supports macOS on Apple Silicon, Linux on x86-64, and Windows on x86-64. The quantizer core is written from scratch in Rust.
DataMoat auto-captures session files from Claude CLI, Codex, Cursor and OpenClaw, normalizes them into a common schema and encrypts with AES-256-GCM. Thesis: AI work traces are private data assets worth preserving.
ha-llm is an open-source Home Assistant integration that exposes a single llm.ask service call routing to OpenAI, Gemini, Claude, OpenRouter, or Ollama backends. It simplifies switching LLM providers in Home Assistant automations without changing service call syntax.
AIFIRST 
