GENERator - A generative DNA model developed by Alibaba AI for Science team and others.

What is a GENERator?

GENERator is a generative genome model developed by the AI for Science team at Alibaba Cloud's Apsara Lab, focusing on the design and generation of DNA sequences. Based on a Transformer decoder architecture, the model features a 98k base pair context length and 1.2 billion parameters, with training data covering 386 billion bp of eukaryotic DNA. It has demonstrated excellent performance in multiple benchmark tests, generating DNA sequences similar to natural protein family structures, and showing optimization potential in tasks such as promoter design.

GENERator's main functions

• DNA sequence generationThe GENERator can generate biologically significant DNA sequences that encode proteins with structures similar to known family proteins. It can successfully generate novel variants of the histone and cytochrome P450 families.
• Promoter designBy fine-tuning, GENERator can design promoter sequences with specific activities to regulate gene expression. Experiments show that the generated promoter sequences differ significantly in activity from the natural samples, demonstrating strong potential for gene expression regulation.
• Genome analysis and annotationIn gene classification and taxonomic tasks, GENERator performs exceptionally well, efficiently identifying gene locations, predicting gene functions, and annotating gene structures.
• Sequence optimizationGENERator has shown significant potential in sequence optimization, providing new tools for synthetic biology and genetic engineering by generating DNA sequences with specific activities through instructions.

GENERator's technical principles

• Transformer decoder architectureGENERator employs a Transformer decoder architecture, achieving efficient sequence modeling through a multi-head self-attention mechanism and a feedforward neural network. The decoder can handle long sequences, avoiding seeing future information during the generation process and ensuring that the generated sequences conform to biological logic.
• Ultra-long context modelingThe model has a context length of 98k base pairs, enabling it to handle complex gene structures. It maintains coherence and biological significance when generating long sequences.
• 6-mer word segmenterGENERator uses a 6-mer segmenter to break down DNA sequences into 6-nucleotide fragments. It outperforms single nucleotide segmenters and BPE segmenters in generation tasks, balancing sequence resolution and context coverage.
• Pre-training strategyThe model was pre-trained on a large-scale dataset containing 386 billion bp of eukaryotic DNA. The pre-training task employed Next Token Prediction (NTP), which learns the semantics of a DNA sequence by predicting the next nucleotide.
• Downstream task adaptationGENERator excels in multiple downstream tasks, including gene classification, taxonomic group classification, and promoter design. Through fine-tuning, the model can generate promoter sequences with specific activities, demonstrating powerful gene expression regulation capabilities.
• Biological verificationThe DNA sequences generated by the model encode proteins with structures similar to those of the natural protein family. The biological significance of the generated sequences was verified by calculating the perplexity (PPL) of the generated sequences using Progen2 and predicting their three-dimensional structures using AlphaFold3.

GENERator's project address

• Project official website:https://generteam.github.io/
• Github repository:https://github.com/GenerTeam/GENERator
• HuggingFace model library:https://huggingface.co/GenerTeam
• arXiv technical paper:https://arxiv.org/pdf/2502.07272

Application scenarios of GENERator

• DNA sequence design and optimizationGENERator can generate biologically meaningful DNA sequences, such as those used for the customization of protein families. It can generate DNA sequences with structures similar to natural protein families, such as variants of the histone and cytochrome P450 families.
• Genome analysis and annotationIn genomics research, GENERator can efficiently identify gene locations, predict gene functions, and annotate gene structures.
• Synthetic Biology and Genetic EngineeringGENERator provides a novel tool for designing and optimizing gene expression regulatory elements, such as promoters and enhancers. It has significant applications in synthetic biology and genetic engineering.
• Precision medicine and drug designBy generating gene sequences associated with specific diseases, GENERator can support precision medicine and drug design. It can be used to design targeted sequences for gene therapy.
• Sequence optimization in biotechnologyGENERator can generate DNA sequences with specific functions through instructions, providing new possibilities for sequence optimization in biotechnology.