LongWriter - A long text generation model jointly developed by Tsinghua University and Zhipu AI.

What is LongWriter?

LongWriter is a long text generation model developed by Tsinghua University in collaboration with Zhipu AI. It can generate coherent text exceeding 10,000 characters, and the project is open source. By analyzing the output limitations of existing large-scale language models, the "LongWriter-6k" dataset was created, successfully expanding the output capabilities of AI models. LongWriter also employs Direct Preference Optimization (DPO) techniques to improve the model's output quality and its ability to adhere to length constraints in instructions.

The main functions of LongWriter

• Generating very long textsLongWriter can generate coherent text of more than 10,000 words, breaking through the limitations of previous AI models in terms of text length.
• Dataset ConstructionBy creating the "LongWriter-6k" dataset, which contains writing samples ranging from 2,000 to 32,000 words, rich long text data was provided for model training.
• AgentWrite methodThe method of automatically constructing ultra-long output SFT data using existing LLMs adopts a divide-and-conquer strategy, which effectively improves the model's ability to generate long texts.
• Direct Preference Optimization (DPO)The DPO technique is used to further optimize the model, improve output quality and the ability to comply with length constraints.

The technical principles of LongWriter

• Long context processing capabilityLongWriter is based on long-context large language models (LLMs) with significantly increased memory capacity, capable of handling historical records of over 100,000 tokens.
• Output length limitation analysisBy analyzing the maximum output length of existing models under different queries, LongWriter identified that the limitation on the model output length mainly stems from the characteristics of the Supervised Fine-Tuning (SFT) dataset.
• Supervised fine-tuning (SFT)LongWriter uses the "LongWriter-6k" dataset in the SFT stage, which allows the model to learn how to generate longer text.

LongWriter's project address

• GitHub repository:https://github.com/THUDM/LongWriter
• HuggingFace model library:https://huggingface.co/THUDM/LongWriter-glm4-9b
• arXiv technical paper:https://arxiv.org/pdf/2408.07055

How to use LongWriter

• Environment configurationEnsure sufficient computing resources are available to run the LongWriter model, including a high-performance GPU and adequate memory.
• Get the modelAccess Github to obtain the open-source code and model of LongWriter.
• Install dependenciesInstall the required dependency libraries and tools according to the project documentation, including deep learning frameworks, data processing libraries, etc.
• Data preparationPrepare long text data suitable for LongWriter processing. Preprocess the data to meet the model's input requirements.
• Model loading: Load the pre-trained LongWriter model, or fine-tune it further based on your own data.
• Writing promptsWrite clear prompts or instructions based on the text content to be generated. The prompts will guide the model to generate specific text.
• Generate textUse the interface or API provided by the model to input prompts and start the text generation process.

Application scenarios of LongWriter

• academic researchLongWriter can assist scholars and researchers in writing long academic papers, research reports, or literature reviews.
• Content creationWriters and content creators can use LongWriter to generate first drafts of novels, screenplays, or other creative writing.
• Publishing industryPublishers can use LongWriter to assist with editing and proofreading, or to automatically generate book content.
• EducationEducators can use LongWriter to generate teaching materials, course content, or learning guides.
• News mediaNews organizations can use LongWriter to quickly generate news reports, in-depth analysis articles, or feature reports.