WebLLM

Overview

WebLLM is a high-performance in-browser LLM inference engine that brings language model inference directly into web browsers. It runs entirely in the browser with WebGPU hardware acceleration and no server support, aiming to enable client-side AI assistants with benefits such as lower cost, personalization, and privacy protection.

Key Features

• Runs language model inference directly in web browsers using WebGPU hardware acceleration, with no server-side processing.
• Provides full OpenAI API compatibility, including streaming, JSON mode, seeding, and preliminary function calling support.
• Natively supports model families such as Llama, Phi, Gemma, Mistral, and Qwen, plus custom models in MLC format.
• Integrates through npm, Yarn, pnpm, or CDN, with modular UI hooks and examples for different use cases.
• Supports streaming chat completions for real-time output in chatbots and virtual assistants.
• Can offload computation to dedicated Web Workers or Service Workers and supports Chrome extension examples.

Use Cases

• Building browser-based chatbots and virtual assistants that generate responses in real time.
• Creating privacy-focused personal AI assistants that run models locally in the client.
• Adding local LLM features to Chrome extensions.
• Prototyping OpenAI-compatible web applications with open-source models.

Getting Started

• Install @mlc-ai/web-llm with npm, Yarn, or pnpm, or import it from a CDN.
• Create an MLCEngine using CreateMLCEngine with a supported model such as Llama-3.1-8B-Instruct-q4f32_1-MLC.
• Send chat messages through engine.chat.completions.create, and use stream: true for streaming responses.

Deployment & Requirements

• A browser environment with WebGPU support and sufficient hardware resources for model inference.
• Model loading requires downloading model files, and the first run can take significant time without a cache.
• The browser Cache API is the default cache backend; IndexedDB and OPFS are also available, with OPFS requiring OPFS support.
• The cross-origin cache backend requires installing and enabling a compatible browser extension.

Before You Adopt

• License: Apache-2.0. Review its terms before using, modifying, or distributing the project.
• The hosted demo is intended for research purposes only and is subject to the model licenses for LLaMA, Vicuna, and RedPajama.
• Initial model loading requires downloading and can take a significant amount of time without cached files, so applications should handle the asynchronous operation.
• Service worker lifecycles are managed by the browser and can be killed at any time without notifying the web application.
• The cross-origin cache backend does not support programmatic tensor-cache deletion; cache clearing is managed by the extension.