DATAGEN

Overview

DATAGEN is an AI-powered data analysis and research platform that uses multiple specialized agents to streamline tasks such as data analysis, visualization, and report generation. It uses LangChain, OpenAI's GPT models, and LangGraph to handle complex research processes and integrates diverse AI architectures.

Key Features

• AI-driven hypothesis generation and validation, automated research direction optimization, and real-time hypothesis refinement.
• Multi-agent system with specialized agents for hypothesis generation, process supervision, visualization, code writing, search, reporting, quality review, and note-taking.
• Robust data cleaning and transformation, scalable analysis pipelines, and automated quality assurance.
• Interactive data visualization, custom report generation, and automated insight extraction.
• Smart memory management with a Note Taker agent for efficient context retention and workflow integration.
• Configurable agent model providers and models through agent_models.yaml, including openai, google, anthropic, ollama, groq, atlascloud, and orcarouter.

Use Cases

• Automating data analysis and complete graphical report writing from user-provided data files.
• Generating, validating, and refining research hypotheses while optimizing research direction.
• Conducting literature and web searches as part of a multi-agent research workflow.
• Producing visualizations, research reports, and quality reviews with revision as needed.

Getting Started

• Clone the repository.
• Create and activate a Conda virtual environment with Python 3.10.
• Install dependencies with pip install -r requirements.txt.
• Rename .env Example to .env and fill all values, including required WORKING_DIRECTORY, CONDA_ENV, and CHROMEDRIVER_PATH.
• Place a data file in the data directory, set the user_input in main.py, and run python main.py.

Deployment & Requirements

• Python 3.10 or higher.
• Conda environment, such as the datagen environment described in the source.
• Required environment variables: WORKING_DIRECTORY, CONDA_ENV, and CHROMEDRIVER_PATH.
• API keys for model providers and web search are optional in the source, but sufficient API credits are needed because the system makes multiple API calls.
• Configuration files are relative to CONFIG_DIRECTORY, which defaults to config/.

Before You Adopt

• License: MIT. Review its terms before using, modifying, or distributing the project.
• The agent system may modify the data being analyzed, so the source recommends backing up data before use.
• The system makes multiple API calls, so sufficient API credits are required.
• A full research run may take time depending on task complexity.
• Firecrawl API key is optional, but missing it may reduce query capabilities.