BigSet - TinyFish, an open-source multi-agent real-time web scraping tool - AiBoss

What is BigSet?

Bigset is an open-source multi-agent system developed by the US startup TinyFish, licensed under the AGPL-3.0 license. Users simply describe the required data in natural language, and the system automatically infers the table structure, dispatches agents to fetch data from the real-time network, performs deduplication and validation, and finally generates an exportable CSV/XLSX structured dataset. The tool supports scheduled refreshes from 30 minutes to weekly, ensuring the dataset remains up-to-date.

BigSet's main functions

Natural Language Table BuildingSimply describe the required data in one sentence, and AI will automatically infer the column names, data types, and primary keys, eliminating the need for manual table structure design.
Multi-agent data collectionThe orchestrator agent discovers the target entity, and the sub-agents capture a single data item in parallel. Each agent can make up to 6 tool calls, automatically completing the entire process from discovery to data filling.
Automatic deduplication and source tracingAutomatic deduplication based on primary key, with each row of data accompanied by its source URL, ensuring that all data is traceable and verifiable.
Refresh periodicallyIt supports automatic updates at five frequencies: 30 minutes, 6 hours, 12 hours, daily, and weekly, keeping the dataset synchronized in real time.
Format exportThe generated results can be directly downloaded in both CSV and XLSX formats, making it easy to integrate with Excel or data analysis tools.
Pre-built datasetIt includes 9 carefully selected public datasets, ready to use out of the box.

BigSet's technical principles

Multi-agent orchestration architectureBigSet is a multi-agent system built on the Mastra framework. Its core consists of an orchestrator and worker agents. The orchestrator is responsible for discovering target entities and generating crawling tasks. The worker agents execute single data crawling in parallel, with each agent having a maximum of 6 tool calls. The Claude Sonnet and Qwen models are uniformly scheduled through Vercel AI SDK and OpenRouter.
Table structure is automatically inferred.After receiving the natural language description, the system uses Claude Sonnet to analyze the semantic intent and automatically infer column names, data types, and primary key constraints. This process does not require manual specification of the schema; the LLM generates a structured table definition based on the entity relationships and data characteristics in the description, which serves as the verification benchmark for subsequent data collection.
Data collection and deduplication verificationThe orchestrator breaks down the target entity into independent tasks and assigns them to sub-agents for parallel fetching. Each sub-agent calls the TinyFish Search / Fetch / Browser API to retrieve data from the real-time network. After the results are returned, the system automatically deduplicates the data based on the primary key and appends the source URL to each row of data to ensure data traceability.
Security protection mechanismTo prevent tooltip injection attacks, the dataset ID is not passed to the LLM via system prompts, but is injected at runtime using a JavaScript closure. The LLM cannot directly access or leak the dataset identifier throughout the process, thus isolating the data collection logic from data permissions.

How to use BigSet

Environmental preparationInstall Docker and Make, and go to the TinyFish, OpenRouter, and Clerk official websites to register an account to obtain API keys.
Cloning repository:implement git clone https://github.com/tinyfish-io/bigset.git Download the project code and... .env.example Copy as .env document.
Configuration key:exist .env The file contains three types of API keys: TinyFish, OpenRouter, and Clerk, to ensure that the service has the permissions for data collection, model invocation, and identity authentication.
Start service:run make dev The system will automatically install dependencies, start the Postgres and Convex databases, and complete the service deployment by executing the command.
Access usageOpen your browser to access localhost:3500After registering and logging in, describe the required data in natural language in the input box, and the system can automatically generate a structured dataset.
Load preset data (optional):implement make seed-public-datasets The command allows you to load nine built-in, carefully selected public datasets with a single click, providing a quick overview of the core features.

BigSet's core advantages

Zero-configuration data collectionStructured datasets can be automatically generated using only natural language descriptions, without the need to write crawlers, selectors, or specify URLs.
Secure isolationThe dataset ID is injected via JS closure, making it inaccessible to the LLM throughout the process, effectively preventing prompt injection attacks.
Open source and self-hostingIt adopts the AGPL-3.0 protocol, can be deployed with Docker with one click, and the data is completely autonomous and controllable.
Enterprise-level infrastructureIt is based on TinyFish's enterprise-grade search and crawling API, which has processed over 40 million agent operations and is stable and reliable.

BigSet's project address

GitHub repositoryhttps://github.com/tinyfish-io/bigset

BigSet's Competitive Product Comparison

Comparison Dimensions	BigSet	Firecrawl
Product Positioning	A multi-agent driven dataset generation platform, from natural language descriptions to exportable structured datasets.	API-first web scraping and crawling platforms convert websites into LLM-ready Markdown or structured data.
Input method	Natural language description requires data topics (e.g., "AI recruitment company information")	Crawle by specifying a URL, or by... `/agent` Endpoints describe requirements using natural language (no URL required).
Data source discovery	AI automatically infers table structure and identifies entity sources without requiring any manual provision of links.	`/agent` It can search and navigate automatically, but in the regular mode, the user needs to specify the starting URL or site map.
Table structure definition	LLM automatically infers the schema (column names, data types, primary keys) and generates a relational dataset.	Supports defining the extraction structure using JSON Schema or natural language suggestions, and outputs a JSON object.
Collection range	Aggregating multi-source data across sites and automatically deduplicating to generate a unified dataset.	Recursively crawl or extract content from a given URL, focusing on content from known websites.
Automation level	Orchestrator + Sub-agent Closed Loop: Discovery → Fetching → Deduplication → Validation → Table Creation → Export, Fully Autonomous Process	Single-link or Agent-based crawling and transformation require external tools for dataset management.
Refresh periodically	Native support (30 minutes / 6 hours / 12 hours / daily / weekly)	Scheduled jobs can be executed periodically, but this requires configuration via API or an external scheduler.
Output format	CSV and XLSX formats (with source URLs, can be used directly in Excel/BI)	Markdown, JSON, HTML, screenshots, links (optimized for LLM/RAG pipelines)
Data deduplication	Automatic deduplication based on primary key ensures dataset uniqueness.	Users need to handle the deduplication logic themselves downstream.

Application scenarios of BigSet

Competitive product monitoringAutomatically tracks competitors' product prices, feature updates, recruitment information, etc., and generates a structured intelligence table that can be refreshed on a regular basis.
Market ResearchCollect company information, financing dynamics, and market share data within a specific industry to quickly build an industry database.
Investment AnalysisData collection to track changes in funding rounds and valuations of stocks, cryptocurrencies, and startups, supporting investment decisions.
Recruitment InformationMonitor job postings, skill requirements, and salary ranges of target companies or industries, and analyze talent market trends.
E-commerce price comparisonIt captures price, inventory, and review data of products from multiple platforms to build a real-time updated price comparison dataset.