PixelRAG - A Berkeley-based open-source visual native RAG framework - AiBoss

What is PixelRAG?

PixelRAG is an open-source, native visual RAG framework developed by Berkeley SkyLab/BAIR. Instead of the traditional approach of extracting web pages into plain text for retrieval, the framework uses a browser to render web pages and PDFs into screenshot tiles. Through LoRA-tuned Qwen3-VL visual vector retrieval, the model can directly read tables, charts, and layouts on the image, solving the pain point of traditional RAGs missing tables and layouts.

PixelRAG's main features

Page rendering (pixelshot)Use headless Chromium to render web pages and PDFs into screenshot tiles, fully preserving tables, charts, and layouts; Windows/macOS automatically calls the system Chrome and supports mixing URLs and local files.
Visual semantic retrievalEncodes screenshots into vectors, supporting text-based and image-based image searches; defaults to FAISS locally, and can be switched to Qdrant on a large scale (supports quantized compression, disk storage, and multi-service sharing).
Pixel direct readingThe screenshots that are found during the search are directly fed as image input to VLM to answer questions, without any text conversion.
Managed search serviceKeyless API + Webpage Demo, with built-in index of 8.28 million Wikipedia pages, search out of the box.
Self-built index pipeline：pixelrag index / embed / build-index / serve The complete set of commands allows you to create libraries for your own PDFs and websites, and can run on a small scale on a Mac (Apple Silicon).
Agent integrationThe Claude Code plugin pixelbrowse, without MCP dependencies, allows Claude to "view" web pages by taking screenshots instead of consuming incomplete HTML.

The technical principles of PixelRAG

Presentation layer—Document as imageUse headless Chromium to take screenshots of web pages, convert PDFs page by page to images and cut them into tiles, so that the visual structure of tables, charts, infographics and other visual structures are preserved as is.
Retrieval Layer - Visual Vector EmbeddingBased on Qwen3-VL-Embedding-2B, LoRA fine-tuning is performed using contrastive learning data, enabling screenshots to be semantically searchable, with queries hitting "which part of the image" rather than "which paragraph of text"; vectors are stored locally in FAISS, and Qdrant can be sliced on a large scale.
Generation Layer – VLM (Visual Model) – Answer Questions Based on DiagramsThe retrieved screenshot tiles are directly fed as pixel input to the visual language model, which reads numbers and layouts directly from the image without any intermediate text conversion.
Scale and efficiencyThe first RAG pipeline to cover the entire Wikipedia page using screenshots; pixel representation also brings new efficiency leverage - reducing screenshot resolution can reduce token cost by up to 3 times without reducing accuracy, and in experiments, it comprehensively surpasses the text RAG baseline, with a maximum improvement of 18.1%.

How to use PixelRAG

Install:implement pip install pixelrag You will then be able to obtain the core tools (Python 3.10+ required).
Rendering page:run pixelshot https://example.com -o ./tiles Render web pages or PDFs as screenshot tiles.
Experience Hosted SearchNo configuration required, just a direct POST request. https://api.pixelrag.ai/search You can search the 8.28 million Wikipedia pages or try it online by opening pixelrag.ai in your browser.
Self-built index:write pixelrag.yaml Configure (specify document path, embedded model, output directory), execute. pixelrag index build After building, use pixelrag serve --index-dir ./my_index --port 30001 Start the search service (it can run on a small scale on a Mac Apple Silicon, but the full Wikipedia index of about 217GB requires GPU computing power).
Connecting to Claude Code:implement uv tool install pixelrag Then install the pixelbrowse plugin (claude plugin marketplace add StarTrail-org/PixelRAG and claude plugin install pixelbrowse@pixelrag-pluginsThis allows Claude to take screenshots and "view" web pages without MCP.
Advanced trainingIf you need to develop your own embedded model, please go to [link/reference]. train/ The table of contents (independent UV project) can be slightly modified according to the README, or the official open-source LoRA weights and training data can be directly reused.

PixelRAG's core advantages

Information losslessIt replaces text extraction with screenshots, preserving the complete table rows, columns, charts, and layout, thus completely solving the pain point of traditional RAGs that seem to have answers but disappear after the text is extracted.
More powerfulThe paper shows that it comprehensively surpasses the text RAG baseline and is also superior on pure text tasks such as NQ and SimpleQA, with a maximum improvement of 18.1% on multimodal and agentic benchmarks.
Lower costPixel representation offers a unique efficiency advantage; reducing screenshot resolution can lower token costs by up to 3 times without compromising accuracy.
Simpler pipelineIt eliminates the complex HTML parsing, cleaning, and segmentation process; simply render a screenshot and store it in the database, significantly reducing the workload.
Scale has been verifiedIt is the first RAG pipeline to cover the entire Wikipedia page in screenshot format, and comes with an 8.28 million-page keyless escrow index available for immediate use.

PixelRAG project address

Project official website：https://pixelrag.ai/
GitHub repository：https://github.com/StarTrail-org/PixelRAG
arXiv technical paper：https://arxiv.org/pdf/2606.28344

Comparison of PixelRAG with similar products

Comparison Dimensions	PixelRAG	ColPali
Producer	Berkeley SkyLab / BAIR / NLP Group (2026)	Illuin Technology (2024)
Base model	Qwen3-VL-Embedding-2B + LoRA fine-tuning	PaliGemma (with a later variant, ColQwen2.5)
Search Object	Primarily webpage screenshots, but also compatible with PDF.	Images of static document pages such as PDF
Embedding granularity	Single vector: Each screenshot corresponds to one vector.	Late-interaction: Retains thousands of patch-level vectors per page.
Storage costs	The single-vector scheme offers high storage efficiency and is easily expandable.	Each page is approximately 256KB, posing a significant challenge for large-scale deployments.
Index size	Verified coverage of the entire Wikipedia page, 30 million screenshots.	Public verification is primarily based on a page count of tens of thousands.
Search accuracy	It outperformed the text baseline across open-domain tasks such as NQ, SimpleQA, and MMSearch, achieving improvements of up to 18.1%.	For layout-intensive documents, fine-grained matching accuracy is superior, consistently placing it at the top of the ViDoRe rankings.
Token efficiency	Reducing screenshot resolution can lower token costs by up to 3 times without compromising accuracy.	No image compression efficiency lever
Engineering Ecology	Full-stack toolchain: rendering, embedding, indexing, managed API, Claude Code plugin	Provide the model and evaluation code; the retrieval service needs to be built independently.

Application scenarios of PixelRAG

Financial Statements and Data Tables Q&AIt directly reads numbers from financial statements and statistical tables, avoiding row and column misalignment caused by text parsing.
Dashboard and Chart SearchThe search function retrieves screenshots containing charts and KPI panels, allowing the model to interpret the images and provide answers.
Infographics and illustrated documentsContent that traditional text indexes cannot cover, such as infographics and flowcharts, can be searched.
Multi-column product pageComplex product comparison pages and news pages should maintain their original layout for search results.
Agent Webpage ReadingBy using Claude Code's pixelbrowse plugin, the Agent can take screenshots and "see" web pages instead of consuming incomplete HTML.