DwarfStar

Overview

DwarfStar is a small native inference engine designed to run a few excellent large language models on consumer hardware. It focuses on specific models including DeepSeek V4 Flash and PRO, GLM 5.2 and 5.3, GLM 5.3 Flash, and Qwen3.8 Flash Next, and supports Metal, CUDA, and ROCm. The project is self-contained and deliberately narrow: it requires GGUF files produced by the project rather than serving as a general GGUF runner.

Key Features

• Native inference engine for selected DeepSeek, GLM, and Qwen models, including experimental vision support for some models.
• Supported backends include Metal, NVIDIA CUDA, and ROCm.
• Self-contained codebase focused on project-produced GGUF files, not a general GGUF runner.
• Integration testing covers model loading, prompt rendering, tool calls, KV state, the HTTP server, and the coding agent.
• Includes tools and data for GGUF, imatrix, quality, and speed.
• Supports SSD streaming, tensor parallelism with RDMA between two 128 GB Macs, pipeline parallelism, and multi-GPU CUDA setups.

Use Cases

• Run capable local models on a MacBook, DGX Spark, or Strix Halo system.
• Use multiple CUDA cards as a multi-user LLM server.
• Use two 128 GB Macs connected with RDMA for tensor-parallel inference.
• Use pipeline parallelism to combine multiple systems and run larger models.

Getting Started

• Clone the repository and enter the ds4 directory.
• Build with make for Metal on Apple Silicon, make cuda-spark for DGX Spark, make strix-halo for Strix Halo, or make cuda-generic for one or more CUDA cards.
• For a first run on a 96 or 128 GB machine, run ./download_model.sh ds4f-q2.
• Run ./ds4 for the default model, ./ds4 -p "Explain Redis streams in one paragraph." for a prompt, or ./ds4-server --ctx 32768 for the server.

Deployment & Requirements

• Metal is the primary target; 64 GB Macs can run Qwen Flash Next, while 96 GB or more is recommended for DeepSeek and GLM. Smaller machines can use SSD streaming.
• NVIDIA CUDA support targets DGX Spark and multi-GPU systems, including Ada Lovelace cards such as L40S.
• ROCm is supported on Strix Halo systems such as the Framework Desktop.
• Downloads go in gguf/. Leave memory for the context and runtime buffers as well as the model.
• Engram tables remain on disk in every mode, so a fast local SSD is needed.

Before You Adopt

• License: MIT. Review its terms before using, modifying, or distributing the project.
• DwarfStar is described as beta quality and fast changing; despite QA runs, instabilities and regressions are possible.
• Model support is intentionally opportunistic, and a supported model may be removed when a better replacement arrives.
• Saved conversations and traces may contain private information.
• Engram tables remain on disk in every mode, so a fast local SSD is required.