WaterCrawl

Overview

WaterCrawl is a self-hosted web application built with Python, Django, Scrapy, and Celery. It combines configurable web crawling and scraping with a search engine, exposed through a REST API and integrations for AI and automation platforms.

Key Features

• Advanced web crawling and scraping with configurable depth, speed, and targeted content.
• Search with basic, advanced, and ultimate depth options.
• Multi-language support with country-specific targeting.
• Asynchronous processing with real-time progress via Server-Sent Events.
• REST API with OpenAPI documentation and client libraries.
• Integrations with Dify, N8N, and other AI or automation platforms.

Use Cases

• Preparing LLM-ready data from websites for AI pipelines.
• Running web research and data collection tasks on self-hosted infrastructure.
• Adding web crawling and search into automation workflows through Dify, N8N, or the REST API.

Getting Started

• Clone the WaterCrawl repository and enter the project directory.
• From the docker directory, copy .env.example to .env.
• Run docker compose up -d.
• Open localhost in a browser to access the application.

Deployment & Requirements

• Docker is used for the documented quick-start deployment path.
• If deploying on a domain or IP other than localhost, update MINIO_EXTERNAL_ENDPOINT, MINIO_BROWSER_REDIRECT_URL, and MINIO_SERVER_URL in the .env file.
• Before production deployment, update the .env file and set up the database, MinIO, and other required services.

Before You Adopt

• License: Other. Review its terms before using, modifying, or distributing the project.
• Non-localhost deployments require MinIO configuration updates, or file uploads and downloads will break.
• Production deployment requires additional configuration of the environment, database, MinIO, and other services.
• Security issues should not be posted on GitHub; the project asks users to email support instead.