EAFormer - An AI text segmentation framework launched by Fudan University

What is EAFormer?

EAFormer is an AI text segmentation framework developed by Fudan University. It excels at finding text in photos and accurately separating it from the background. Even with blurry text edges or complex backgrounds, EAFormer performs admirably. It's extremely useful for anyone wanting to remove or edit text from images. EAFormer is a smart assistant that makes manipulating text in images easy.

EAFormer's main functions

• Find textEAFormer can quickly find all the text in photos or pictures.
• Outline cornerIt can not only find the characters, but also depict the edges of the characters with great accuracy, even the crooked parts.
• Change backgroundIf you want to remove the text from an image and replace it with a new background, EAFormer makes the text look like it never existed.
• Learn new tricksIf EAFormer encounters new or different languages, it can quickly learn and improve its recognition capabilities.

EAFormer's technical principles

• Text edge extractorThe Canny algorithm is used to detect edges in images, and a lightweight text detection model is combined to filter the edges of non-text regions, retaining only the edge information of text regions.
• Edge Guide EncoderBuilt on the SegFormer framework, it incorporates edge information during the encoding process through symmetrical cross-attention layers, enhancing the model's ability to perceive text edges.
• MLP decoderThe method employs a multilayer perceptron (MLP) layer to fuse features and predict the final text mask, thereby achieving accurate segmentation of text regions.
• Loss function designThe method uses two types of cross-entropy loss, text detection loss and text segmentation loss, for optimization. By balancing the weights of the two through hyperparameters, the complexity of hyperparameter selection is simplified.
• Dataset relabelingTo address the annotation quality issues of datasets such as COCO_TS and MLT_S, we re-annotated them to ensure the reliability of the evaluation results and the accuracy of model training.
• Feature fusion strategyIn the edge-guided encoder, the designed symmetrical cross-attention mechanism fuses edge information only in the first layer, avoiding the performance degradation that may result from fusing edge information in all layers.
• Lightweight text detectorUsed in text edge extractors, including a ResNet-based backbone network and an MLP decoder, to extract text region features and assist in edge filtering.

EAFormer project address

Application scenarios of EAFormer

• Scene text recognitionIt identifies and segments text in natural scenes or images for information extraction or data mining.
• Image editingIt assists image editing software in precisely erasing or replacing text in images while maintaining the naturalness and coherence of the background.
• Ad blockingAutomatically detect and block advertisements or other unwanted text in video streams or images.
• Copyright protectionIt helps identify and protect copyrighted text, preventing unauthorized copying or distribution.
• Document processingText recognition during automated document scanning and digitization improves the efficiency and accuracy of document processing.