AB
AB
AiBoss

All Categories

project

DataClaw - an open-source AI dialogue data export tool that converts data into standard training sets with a single click.

DataClaw is an open-source AI dialogue data export tool developed by Peter O'Malle. It can automatically capture the complete dialogue history between users and AI programming assistants such as Claude Code, Codex CLI, and Gemini CLI, and convert it into...

What is DataClaw?

DataClaw is an open-source AI dialogue data export tool developed by Peter O’Malle. It automatically captures the complete dialogue history between users and AI programming assistants such as Claude Code, Codex CLI, and Gemini CLI, converting it into a structured JSONL training dataset. This dataset can be published to the Hugging Face platform with a single click for community use. The tool includes built-in PII (Personally Identifiable Information) detection and sensitive key filtering functions, automatically removing passwords, API keys, and other private information before sharing to ensure data security.

Main functions of DataClaw

  • Automatic capture of conversation historyIt supports exporting complete conversation records from mainstream AI programming assistants such as Claude Code, Codex CLI, and Gemini CLI, eliminating the need for manual copying and pasting.
  • Privacy-friendly intelligent desensitizationIt has a built-in PII (Personally Identifiable Information) detection engine that automatically identifies and removes sensitive content such as passwords, API keys, and email addresses to ensure data security.
  • Structured format conversionConvert the original dialogue into standard training data formats such as JSONL, making it easier to use for fine-tuning large language models.
  • One-click posting to Hugging FaceIt supports pushing the processed dataset directly to Hugging Face Hub for download and use by the open-source community.
  • Multi-platform data integrationIt is compatible with the data formats of various AI programming tools, enabling unified management of cross-platform dialogue data.
  • Open source and customizableDeveloped using Python, the code is completely open source, and users can customize data processing rules and de-identification strategies according to their needs.

DataClaw's Technical Principles

  • Local file system monitoringBy monitoring local dialogue log files (such as JSON or SQLite databases) generated by tools like Claude Code and Codex CLI, the system can capture complete records of user interactions with AI in real time.
  • PII detection and regular matchingIt employs rule-based regular expressions and keyword matching algorithms to identify and filter sensitive information such as API keys, passwords, email addresses, and ID card numbers, ensuring de-identification.
  • Dialogue Structured ParsingIt parses unstructured natural language dialogues into a standard JSONL format containing fields such as role (user/assistant), content, and timestamp, conforming to OpenAI's fine-tuning data specifications.
  • Hugging Face API IntegrationEnables one-click dataset uploads via Hugging Face Hub's Python SDK, automatically handling authentication, repository creation, and version management.
  • Incremental synchronization mechanismSupports incremental data capture, exporting only newly added dialogue content to avoid redundant processing and full data overwrite.
  • Cross-platform adaptation layerThe built-in adapter provides unified conversion for the different formats of various AI tools (such as Claude's XML logs and Codex's JSON format).

DataClaw's project address

  • GitHub repositoryhttps://github.com/peteromallet/dataclaw

Application Scenarios of DataClaw

  • Open source model fine-tuningIt provides developers with high-quality, real-world programming dialogue data to fine-tune open-source code models such as CodeLlama and DeepSeek-Coder, improving their performance on specific programming languages or frameworks.
  • Research on AI Programming AssistantsResearchers can use the collected dialogue data to analyze user interaction patterns with AI programming assistants, optimize prompting engineering strategies, or evaluate model performance.
  • Construction of Educational Training DatasetsProgramming education institutions can compile dialogue data into a teaching case library to train students on how to effectively collaborate with AI in programming.
  • Competitive Model DistillationOther AI companies or research teams can use publicly available dialogue data as a distillation source to train smaller, more efficient programming-specific models.
  • Data Democratization MovementSupport the open-source community in countering the data-locked strategies of large AI companies and promote the open sharing and fair use of AI training data.
Back