TensorFold

Overview

TensorFold is a local inference engine for large language models, distributed as an open-source Apache-2.0 project. It runs open checkpoints on Apple Silicon and NVIDIA GPUs using native Metal or CUDA kernels and serves them without requiring Python or MLX. The project provides a CLI server, model pull and inspection commands, OpenAI-compatible and Anthropic API surfaces, and a desktop workspace for Apple silicon.

Key Features

• Runs open language models locally on Apple Silicon and NVIDIA GPUs using native Metal or CUDA inference, served from a Zig binary without Python or MLX.
• Exposes OpenAI-compatible and Anthropic API surfaces, including chat completions, completions, Responses, Anthropic Messages, and model discovery through /v1/models.
• Supports parallel decode, prompt reuse, and connected Macs so drafted tokens, retained prompts, and multiple machines work through the same engine.
• Provides CLI model workflows to pull Hugging Face checkpoints, list cached models, inspect family, format, context, and memory floor, and resume interrupted downloads.
• Includes a desktop workspace for Apple silicon that bundles the local engine, model downloads, hardware recommendations, live generation metrics, coding workers, and document editing.
• Offers optional server capabilities including context compaction, Cortheon research mode, parallel request limits, API keys, metrics, dashboard, and experimental Living Weights.

Use Cases

• Run open models on GPUs you own instead of relying on hosted inference, using native Metal or CUDA acceleration.
• Serve a local checkpoint to OpenAI-compatible or Anthropic-style clients from one CLI server.
• Use model pull and inspection commands to manage quantized checkpoints and check hardware fit before serving.
• Run larger models across connected Apple silicon machines or multiple NVIDIA GB10 systems where the specific model recipe supports it.

Getting Started

• On Apple Silicon, install TensorFold with Homebrew, pull a supported checkpoint such as mlx-community/Qwen3.5-2B-MLX-4bit, and serve it with a chosen name and context size.
• On Linux with an NVIDIA GPU, download the release archive for the host CPU, verify its SHA-256 file, extract it, add its bin directory to PATH, then pull and serve a checkpoint.
• After the server is running, send an OpenAI-compatible chat completion request to the local server from another terminal; the API also serves completions, Responses, and Anthropic Messages.
• A desktop workspace is available for Apple silicon as a beta DMG, or it can be built from the app directory.

Deployment & Requirements

• The Mac app beta targets Apple silicon and macOS 13.3 or newer.
• Linux NVIDIA deployments use the linux-aarch64 archive on GB10 and linux-x86_64 on x86_64 hosts, require glibc 2.28 or newer, and need an NVIDIA driver providing libcuda.so.1.
• Keep the Linux release archive's directories together after extraction.
• Memory beyond model weights is needed for context, parallel requests, prompt caches, and drafters; check the checkpoint memory floor with tensorfold info REPO before serving large models.
• On Windows with an NVIDIA GPU, the documented path is the native Windows build guide rather than a ready release archive.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• The TensorFold desktop app is currently a beta release.
• Model weights carry their own licenses separate from the engine.
• Some model backends are not yet included in a release binary.
• Benchmark figures are individual release measurements and should not be treated as like-for-like comparisons.