Firecrawl

Overview

Firecrawl is the web data API to search, scrape, and interact with the web at scale. It helps find sources, extract content, and turn it into clean Markdown or structured data that AI agents can use. The project is open source and also available as a hosted service, with core endpoints for Search, Scrape, and Interact plus Agent, Crawl, Map, and Batch Scrape.

Key Features

• Search the web and get full page content from results.
• Scrape any URL into Markdown, HTML, screenshots, or structured JSON.
• Interact with a scraped page using AI prompts or code.
• Use the Agent endpoint to describe the data needed and let the AI agent search, navigate, and retrieve it.
• Crawl all URLs of a website, map all site URLs instantly, and batch scrape thousands of URLs asynchronously.
• Connect to AI agents and MCP clients, with SDKs for Python, Node.js, Go, Java, Elixir, Rust, Ruby, .NET, and PHP.

Use Cases

• Build AI agents that need real-time web data in clean Markdown or structured JSON.
• Search the web and retrieve full page content for research, monitoring, or RAG pipelines.
• Scrape single pages or crawl entire websites into LLM-ready formats.
• Automate data gathering with a prompt when URLs are not known in advance.

Getting Started

• Sign up on the Firecrawl website to get an API key.
• Install an SDK, such as pip install firecrawl-py for Python or npm install firecrawl for Node.js.
• Create a Firecrawl client with your API key and call endpoints such as search, scrape, agent, crawl, map, or batch scrape.
• For MCP, configure an MCP server with npx -y firecrawl-mcp and set the FIRECRAWL_API_KEY environment variable.

Deployment & Requirements

• A Firecrawl API key is required for the hosted service; sign up on the Firecrawl website to obtain one.
• The project is open source and can be run locally; the contributing guide provides local setup instructions.
• The project can be self-hosted, with a separate self-hosting guide.
• The cloud version includes additional features compared with the open-source version.

Before You Adopt

• License: AGPL-3.0. Review its terms before using, modifying, or distributing the project.
• Users are advised to follow applicable privacy policies and terms of use, and Firecrawl respects robots.txt directives by default.
• Use of Firecrawl is subject to compliance with the stated scraping and privacy conditions.
• When using the Agent endpoint, sending both the model and effort fields results in a 400 error.