LLM Council - Karpathy, an open-source multi-model collaboration framework - AiBoss

What is the LLM Council?

LLM Council is Andrej Karpathy open sourceMulti-model collaboration frameworkBy using OpenRouter, user questions are simultaneously distributed to multiple large models (such as Claude, GPT, Gemini, etc.) to answer independently. Each model anonymously evaluates and scores each other, and finally, a designated Chairman LLM synthesizes all opinions and outputs the optimal answer, thus realizing multi-model cross-validation and collective decision-making.

Main functions of LLM Council

Parallel response using multiple modelsA single question can trigger multiple LLMs to generate answers independently, and tabs can be used to view the original responses of each model.
Anonymous cross-reviewEach model anonymously reviews the responses of other models and ranks them according to accuracy and insight, avoiding bias towards itself.
Chairman Model SynthesisThe designated Chairman LLM will compile all initial drafts and review comments, and output the final unified answer.
Local session managementThe conversation history is stored locally as a JSON file, allowing you to review and continue the discussion.
Visualizing the three-stage processThe web interface clearly displays the complete workflow from "first draft → review → final draft".

LLM Council's Technical Principles

Backend architectureBuilt on FastAPI (Python 3.10+), it uses asynchronous httpx concurrent calls to the OpenRouter API to implement multi-model parallel requests.
Front-end technologySupports the React + Vite technology stack, and outputs Markdown content using the react-markdown rendering model.
Anonymous review mechanismStage 2 anonymizes the identities of each model to ensure the objectivity and fairness of the review process and eliminate brand bias.
Three-stage production lineStage 1: Collect initial drafts of each model → Stage 2: Distribute anonymous responses for peer review and ranking → Stage 3: The chairperson synthesizes the final draft of the model.
Local data storageSession data is stored in JSON file format. data/conversations/ Directory, no external database required.

How to use LLM Council

Cloning repository:implement git clone https://github.com/karpathy/llm-council.git Download the project code.
Install dependenciesBackend operation uv syncFront end enters frontend Directory Execution npm install.
Configure API keyCreate in the project root directory .env File, fill in OPENROUTER_API_KEY=sk-or-v1-....
Custom Model(Optional): Edit backend/config.py Revise COUNCIL_MODELS Lists and CHAIRMAN_MODEL.
Launch application:implement ./start.sh One-click start, or start the backend and frontend development servers separately.
Start conversationAccess via browser http://localhost:5173Entering a question will trigger the multi-model advisory committee process.

LLM Council's core strengths

Eliminating single-model biasThrough multi-model cross-validation and cross-checking, illusions, blind spots, and inherent biases are significantly reduced.
The quality of the answers has been significantly improved.The dual mechanism of review and screening plus chairman's summary results in a more comprehensive and reliable output than a single model.
Fully local deploymentThe code is open source, data is stored locally, and conversation privacy does not depend on any third-party platform.
Model combination is highly flexibleOpenRouter allows for the combination of any model, and the community has expanded support for local Ollama model integration.
Lightweight and quick setup99% AI-assisted coding (Vibe Code), extremely simple dependency, local deployment can be completed in minutes.

LLM Council project address

GitHub repositoryhttps://github.com/karpathy/llm-council

LLM Council's Competitive Product Comparison

Dimension	LLM Council	AutoGen	CrewAI
Product Positioning	Multi-model answer integration and review tool	Conversational Multi-Agent Orchestration Framework	Role-driven task collaboration framework
Collaboration mode	Parallel responses + anonymous peer review + chair arbitration	Group chat dialogue + consensus negotiation	Role division of labor + sequential/hierarchical execution
Model source	Cross-vendor (OpenRouter aggregation)	Single or isomorphic model	Single or isomorphic model
Anonymous review	Native support	Not supported	Not supported
Peer review and scoring	Native support	Not supported	Not supported
Chairman's Summary	Native support	Custom implementation required	Manager/Agent can be replaced
Deployment method	Completely local	Local/Cloud	Local/Cloud
Data privacy	Data is stored entirely locally	Depending on the deployment method	Depending on the deployment method
Learning curve	Extremely low (one-click start)	Intermediate level (requires understanding of dialogue paradigms)	Low (character configuration is intuitive)
Applicable Scenarios	Question-and-answer decision-making, content review, and model evaluation	Code generation, scientific research, debate	Content creation, market research, and process automation

Application scenarios of LLM Council

Decision Making for Complex ProblemsImportant decision-making scenarios that require cross-validation of multiple perspectives, such as technology selection and business strategies.
Academic research assistanceWhen reading papers or books, obtain different interpretations and in-depth analyses of the same concept from multiple models.
Content quality reviewIt enables multiple models to review articles, code, or solutions, and automatically selects the best version.
Horizontal evaluation of model capabilitiesCompare the quality of different LLMs in answering the same question to assist the team in model selection.
Creative brainstormingUse a multi-model perspective to inspire more diverse creative solutions and avoid a single mindset.