Skip to main content
AdOpenFree logoPromote your productReach more potential users and drive product growth and revenue.Advertise
Favicon of llama.cpp

llama.cpp

Free Listing

LLM inference in C/C++ with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud.

Visit ggml-org/llama.cpp

Overview

llama.cpp enables LLM and VLM inference with minimal setup and state-of-the-art performance on a wide range of hardware, locally and in the cloud. It is built on top of the ggml library.

Key Features

  • Plain C/C++ implementation without any dependencies
  • Optimized for Apple silicon via ARM NEON, Accelerate, and Metal frameworks
  • Integer quantization from 1.5-bit to 8-bit for faster inference and reduced memory use
  • Custom CUDA kernels for NVIDIA GPUs, with AMD via HIP and Moore Threads via MUSA
  • Vulkan and SYCL backend support
  • CPU+GPU hybrid inference for models larger than VRAM

Use Cases

  • Run LLM and VLM models locally from Hugging Face
  • Launch an OpenAI-compatible API server
  • Use the built-in web UI for interactive sessions
  • Accelerate large models with hybrid CPU+GPU inference

Getting Started

  • Visit https://llama\.app and follow the instructions
  • Run with Docker (see Docker documentation)
  • Download pre-built binaries from the releases page
  • Build from source by cloning the repository (see build guide)
  • Example: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

Deployment & Requirements

  • Supports a wide range of hardware backends including CPU, NVIDIA GPU (CUDA), AMD GPU (HIP), Apple Silicon (Metal), Intel GPU (SYCL), Vulkan, WebGPU, and more
  • No external dependencies for the core implementation
  • Can be deployed via Docker or pre-built binaries

Before You Adopt

  • License: MIT. Review its terms before using, modifying, or distributing the project.
  • The OpenVINO backend is marked as in progress, indicating incomplete support.

Comments

Sign in to leave a comment.

More like llama.cpp

Favicon of Edge0

Edge0

Free ListingStars: 11K

Device-native foundation models for on-device AI

Language Models

Edge0 provides device-native foundation models and an open-source streaming MoE inference framework, running large sparse models such as 145B on a laptop and 35B on a phone with SSD-streamed weights, offline on-device processing, and zero

Favicon of kimi-k3-in-c

kimi-k3-in-c

Free ListingStars: 10K

2.78T-parameter Kimi K3 inference in portable C99

Language Models

Runs Kimi K3, a 2.78-trillion-parameter model, on a single CPU in 8.24 GB of RAM by streaming the model from disk instead of holding it in memory. Portable C99 with no BLAS, framework, or GPU.

Favicon of nanochat

nanochat

Free ListingStars: 60.1K

Minimal experimental harness for training LLMs

Language Models

nanochat is a minimal, hackable harness for training LLMs end to end on a single GPU node, covering tokenization, pretraining, finetuning, evaluation, and inference. It is pitched as the best ChatGPT that $100 can buy.