AB
AB
AiBoss

All Categories

project

SurfSense - an open-source AI research assistant that seamlessly connects your personal knowledge base to global data sources.

SurfSense is an open-source AI research tool, similar to NotebookLM and Perplexity, but with greater extensibility. It can integrate with various external data sources (such as search engines, Slack, Notion, YouTube, GitHub, etc.).

What is SurfSense?

SurfSense is an open-source AI research tool, similar to NotebookLM and Perplexity, but with greater extensibility. It integrates with various external data sources (such as search engines, Slack, Notion, YouTube, GitHub, etc.), supports users uploading files in various formats, and integrates content into their personal knowledge base. SurfSense provides powerful search capabilities and natural language interaction, allowing users to quickly find and reference saved content.

SurfSense's main functions

  • Powerful searchCombines semantic and full-text search to quickly find content in the knowledge base.
  • Multi-format supportSupports uploading more than 50 file formats (such as documents, images, etc.).
  • Natural Language InteractionUsers ask questions in natural language, and the system generates cited answers based on the content in the knowledge base, resulting in a natural and smooth interactive experience.
  • External data source integrationIt integrates with various external data sources such as search engines, Slack, Notion, YouTube, and GitHub to broaden the channels for obtaining information.
  • Privacy and on-premises deploymentSupports local LLM, ensures data privacy, and can be self-hosted.
  • Browser extensionBrowser extensions allow users to easily save webpage content, especially pages that require login, for later organization and retrieval.
  • Document ManagementIt facilitates the management of uploaded files and supports multi-document interaction.

SurfSense's technical principles

  • RAGBased on vector embedding and full-text search technologies, this system quickly retrieves the most relevant document fragments from a knowledge base to the user's query. The retrieved fragments are then used as context input into an LLM (Local Level Management) to generate a natural language response. Combining semantic search (based on vector embedding) and full-text search (based on keyword matching), Reciprocal Rank Fusion (RRF) is used to optimize the relevance of the search results.
  • Vector embedding and indexingThis feature utilizes pgvector (a vector extension for PostgreSQL) for efficient vector similarity operations. Document content is embedded in a vector space, facilitating the rapid retrieval of semantically relevant segments.
  • Hierarchical IndexBased on a hierarchical index structure (2-layer RAG architecture), it improves retrieval efficiency and accuracy. The first-layer index is used to quickly filter relevant documents, and the second-layer index is used to further refine the search results.
  • Backend technology stackA modern API framework built with FastAPI supports high-performance web services. The database uses PostgreSQL, combined with pgvector to provide vector search capabilities. LLM integration is based on LangChain and LiteLLM, supporting multiple language models.
  • Front-end technology stackBuild user interfaces using Next.js and React to deliver a smooth interactive experience. Provide customized UI design and animation effects based on Tailwind CSS and Framer Motion.
  • Browser extensionUse Plasmo to build browser extensions that support cross-browser functionality, allowing users to easily save web page content.

SurfSense project address

Application scenarios of SurfSense

  • Personal knowledge managementIt helps users organize notes, documents, etc., quickly retrieve knowledge base content, and improve the efficiency of personal knowledge management.
  • academic researchIt assists in literature retrieval and data organization, generates research reports, and improves the efficiency and quality of academic research.
  • Enterprise Knowledge SharingEnterprises can upload internal materials, promoting team collaboration and knowledge flow, and improving the enterprise's knowledge management capabilities.
  • Content creationIt helps creators gather creative inspiration, organize reference materials, assist in writing, and stimulate creative inspiration.
  • Information gatheringUsers can save webpage content, integrate external information, improve the efficiency of information organization, and better meet various information needs.
Back