Scrapling

Overview

Scrapling is a Python web scraping framework that combines fast HTTP requests, stealth browsers, powerful parsing, and concurrent crawling in one library. It supports static pages, JavaScript rendering, website interaction, API capture, reusable sessions, proxy rotation, retries, pause/resume, and built-in data exports. It can be used from Python, the terminal, or an AI agent through its MCP server and Agent Skill.

Key Features

• Spiders provide a Scrapy-like API with concurrent crawling, multi-session support, pause and resume, streaming, blocked-request detection, AutoThrottle, and optional Robots.txt compliance.
• Fetchers support fast HTTP requests, stealthy browser automation, JavaScript rendering, persistent sessions, proxy rotation, domain and ad blocking, and background API capture.
• Parsing supports CSS and XPath selectors, text and regex search, adaptive element tracking, similar-element matching, and clean Markdown output.
• Built-in exporters produce JSON, JSONL, CSV, and XML results, with ready-made spider templates for common crawling patterns.
• AI features include an MCP server for agents, an Agent Skill, and RAG-ready Markdown generation.
• Developer tooling includes an interactive shell, CLI usage, auto selector generation, type hints, and a Docker image.

Use Cases

• Scraping static pages or rendering JavaScript to extract structured data from websites.
• Building concurrent crawlers with sessions, proxy rotation, retries, pause and resume, and data exports.
• Capturing API responses behind web pages for data extraction.
• Enabling AI agents to browse and extract data through the MCP server or preparing web content as Markdown for RAG systems.

Getting Started

• Scrapling requires Python 3.10 or higher.
• Install the base package with pip install scrapling.
• For fetchers or spiders, install optional dependencies with pip install "scrapling[fetchers]" and then run scrapling install.
• Additional extras include scrapling[ai], scrapling[rag], scrapling[shell], and scrapling[all].

Deployment & Requirements

• Python 3.10 or higher is required.
• The base installation includes the parser engine only; fetchers and spiders require optional dependencies.
• Browser fetchers require running scrapling install to download browsers and system and fingerprint dependencies; HTTP requests do not need a browser installation.
• A Docker image is available as pyd4vinci/scrapling and ghcr.io/d4vinci/scrapling:latest with all extras and browsers.

Before You Adopt

• License: BSD-3-Clause. Review its terms before using, modifying, or distributing the project.
• The base pip package includes only the parser engine and its dependencies; fetchers and spiders require installing optional dependencies first.
• Browser-based fetchers require the scrapling install step, while HTTP requests do not require a browser installation.