AB
AB
AiBoss

All Categories

project

WebSSL - A series of visual self-supervised learning models developed by Meta in collaboration with New York University and other institutions.

WebSSL (Web-scale Self-Supervised Learning) is a series of visual self-supervised learning (SSL) models developed by institutions such as Meta and New York University. These models train visual models on massive amounts of web data (such as billions of images)...

What is WebSSL?

WebSSL (Web-scale Self-Supervised Learning) is a series of visual self-supervised learning (SSL) models developed by institutions such as Meta and New York University. It trains visual models on massive amounts of web data (such as billions of images) without requiring language supervision. WebSSL includes several model variants, such as Web-DINO and Web-MAE, with parameter sizes ranging from 300 million to 7 billion. The models perform exceptionally well in multimodal tasks such as visual question answering (VQA), OCR, and graph understanding, even surpassing language-supervised models (such as CLIP). The core advantage of WebSSL lies in its ability to utilize massive amounts of data and its sensitivity to data distribution. By filtering image data containing more text, it significantly improves OCR and graph understanding capabilities.

Main functions of WebSSL

  • No language supervision requiredBased on training with large-scale image data, it can learn effective visual representations without language supervision.
  • Excellent performance in multimodal tasksIn multimodal tasks such as visual question answering (VQA), OCR, and graph understanding, its performance matches or even surpasses that of language-supervised models (such as CLIP).
  • Data filtering improves the performance of specific tasksImprove OCR and chart understanding capabilities by filtering image data that contains more text.
  • The model and data are highly scalable.As the model capacity and training data increase, performance continues to improve.

The technical principles of WebSSL

  • Self-Supervised Learning (SSL)Visual representations are learned from large-scale unlabeled image data using self-supervised learning methods, such as contrastive learning or masked image modeling. Contrastive learning learns semantic representations of images by bringing different augmented views of an image closer together and pushing them further apart. Masked image modeling learns the local and global structure of an image by predicting the masked portions within it.
  • Large-scale data trainingTraining with large-scale network data provides the model with rich learning material due to the diversity and scale of the data. By increasing the scale of the training data, the model can learn broader and more complex visual concepts.
  • Model extensionExpanding the model's parameter size (from 300 million to 7 billion parameters) enhances the model's learning and representation capabilities. Large-scale models can capture more complex visual patterns and semantic information, performing exceptionally well in multimodal tasks.
  • Data filteringFiltering image data containing more text (such as charts, documents, etc.) improves OCR and chart understanding capabilities. Data filtering strategies allow the model to focus on learning visual features related to text, resulting in better performance in relevant tasks.
  • Evaluation of multimodal tasksVisual Question Answering (VQA) is used as the primary evaluation framework, covering multiple task categories (such as general, knowledge, OCR and chart, and visual center tasks). This comprehensive evaluation method better reflects the model's performance in real-world applications.

WebSSL project address

WebSSL application scenarios

  • Multimodal visual question answeringIt can be used in scenarios such as intelligent customer service and educational assistance to help understand image content and answer related questions.
  • OCR and Chart UnderstandingIn document processing and data analysis, it accurately identifies text and chart information in images.
  • Image classification and segmentationIt is used in medical image analysis and autonomous driving for accurate image recognition.
  • Visual content recommendationUsed in image or video content recommendation systems to recommend relevant content based on user preferences.
  • Robot vision and environmental perceptionIt helps robots better understand their surroundings and improves their autonomy and interaction capabilities.
Back