Abstract
Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm, effectively functioning as solvers for independent requests rather than assistants in continuous dialogue. To bridge this gap, we present ChatUMM. As a conversational unified model, it excels at robust context tracking to sustain interleaved multimodal generation. ChatUMM derives its capabilities from two key innovations: an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow, and a systematic conversational data synthesis pipeline. This pipeline transforms a diverse set of standard single-turn datasets into fluid dialogues through three progressive stages: constructing basic stateful dialogues, enforcing long-range dependency resolution via "distractor" turns with history-dependent query rewriting, and synthesizing naturally interleaved multimodal responses. Extensive evaluations demonstrate that ChatUMM achieves state-of-the-art performance among open-source unified models on visual understanding and instruction-guided editing benchmarks, while maintaining competitive fidelity in text-to-image generation. Notably, ChatUMM exhibits superior robustness in complex multi-turn scenarios, ensuring fluid, context-aware dialogues.
Teaser
Figure 1: Examples of ChatUMM demonstrating diverse conversational capabilities.
Method
Figure 2: Overview of ChatUMM (Interleaved Multi-turn Training). The model processes a serialized stream of interleaved text and image tokens using special tokens as structural delimiters. |im_s| and |im_e| encapsulate textual segments, while |v_s| and |v_e| enclose visual content. Crucially, specific token transitions serve as explicit intent signals: predicting |v_s| immediately following |im_e| triggers image generation, whereas predicting |im_s| after |v_e| instructs the model to initiate text generation. The |end| token marks the completion of a turn, and |NTP| represents the text tokens generated via next token prediction. For visual generation, the image of the current turn (e.g., Turn-1 and Turn-2) is processed as noised VAE latents (striped blue), supervised by the flow matching loss ℒMSE. To retrieve context from the dialogue history (e.g., Turn-1 referenced during Turn-2), the model attends to historical text tokens, clean VAE latents (solid blue), and ViT tokens (yellow). Concurrently, the standard cross-entropy loss (ℒCE) is applied to |NTP| and special tokens to supervise text generation and intent prediction.
Data Pipeline
Figure 3: Overview of our data synthesis pipeline. We transform standard single-turn datasets into fluid, stateful dialogues through three progressive stages. (a) Basic multi-turn construction: We leverage atomic LLM-powered operations to convert single-turn samples into basic stateful dialogues. (b) Independent single-turn insertion: To enforce long-range dependency resolution, unrelated "distractor" turns are inserted into the flow. The subsequent query is rewritten to be explicitly history-dependent (<query-dep>), teaching the model to maintain robust context tracking across a noisy history. (c) Interleaved output generation: We evolve the output modality to sustain interleaved multimodal generation. An LLM generates a relevant Q&A pair based on the image of the final turn, transforming the interaction into a continuous text-image stream (i.e., User: <query-final><Q>, Assistant: <image><A>).
Atomic Operations
Figure 4: Visualization of selected atomic operations. Top: caption2query converts image captions into user queries. Bottom: query2dep_q transforms user queries into specific, explicit instructions that clearly identify the target image or subject.
Multimodal Understanding
Figure 5: Qualitative examples of multimodal understanding capabilities. ChatUMM demonstrates versatility in diverse tasks, including chart interpretation, celebrity recognition with knowledge retrieval, explanation of visual humor, and detailed image captioning.
Text-to-Image
Figure 6: Qualitative examples of text-to-image. ChatUMM excels at generating high-fidelity images across various styles, such as photorealistic interiors, traditional Chinese paintings, and Gilt-style portraits. It accurately renders specific text elements and lighting details described in long, detailed prompts. Notably, the bottom example highlights the interleaved generation capability: the model generates a snowy landscape and provides a step-by-step skiing tutorial in response to the user's request.
Image Editing
Figure 7: Qualitative examples of instruction-guided image editing. ChatUMM performs precise text replacement within images and integrates objects into scenes. Notably, the right example showcases the model's interleaved generation capability: it executes a style transfer while simultaneously explaining the cultural meaning of the flower in the provided image.
Interleaved Dialogue
Figure 8: Example of robust multi-turn capabilities. ChatUMM seamlessly interleaves diverse tasks, including image generation, visual reasoning, and factual Q&A, within a continuous conversational flow. The final turn highlights its long-range image editing capability: the model executes the instruction via accurate retrieval of the initial image across multiple "distractor" turns.
Basic Multi-Turn Interactions
Figure 9: Basic multi-turn interactions. Top: Subject consistency during editing. Middle: Context-aware generation from text history and merging prior subjects into a single scene. Bottom: An interleaved sequence of knowledge retrieval, generation, and editing.
Advanced Multi-Turn Dialogues
Figure 10: Advanced multi-turn dialogues. Left: Targeted subject editing. The model correctly edits the "Knight" from the first turn, effectively bypassing the unrelated "Princess" generated in the second turn to perform history-dependent editing. Right: Long-range subject-driven generation. The model combines two characters from the initial turns, effectively filtering out a sequence of "distractor" turns (e.g., Mummy, Fauvism painting) to execute the final composition.
Benchmark Results
