Multimodal
A model that accepts or produces more than one type of data, such as text, images, audio or video. Most current frontier models take text and images in but answer only in text; speech, image and video output usually come from separate specialized models.
A multimodal model can take in, and sometimes produce, data beyond text: images, documents, audio and video. A vision-capable model handles a screenshot and a question in the same request and reasons over both together, instead of passing the image through a separate captioning model first.
What current models accept
- Text and images: The baseline for closed frontier models. Anthropic says all current Claude models, including Claude Opus 5.5, support text and image input with text output, and they also read the text, charts and tables in PDFs. OpenAI lists text and image input with text output for GPT-6.1 Sol and GPT-6.1 Terra. Open-weight flagships increasingly match this: Mistral describes Mistral Large 4 as a general-purpose multimodal model with a vision encoder. Some new open models are still text-only, so check the model card.
- Video: Google's Gemini 4 Argon and Gemini 4 Hydrogen take video files directly, and Google reports 91.7% for Argon on the LVBench long-video benchmark. Moonshot's Kimi K3.1 Pro also accepts video files. Claude and GPT-6.1 Sol do not, so video has to be sent as sampled frames.
- Audio: Gemini 4 Argon and Gemini 4 Hydrogen accept audio files but reply in text. Real-time voice runs on dedicated speech models: OpenAI's GPT-Live-1 is the full-duplex model that takes and returns audio and text through OpenAI's Live endpoint (ChatGPT already runs the newer GPT-Live-1.1, which OpenAI says is coming to the API), and Google's Gemini 3.8 Live is the native-audio model for its Live API.
- Documents: PDF input is common. Google lists PDF alongside text, image, audio and video for Gemini 4 Argon. Reliability on complex layouts still varies.
What current models produce
General-purpose chat models answer in text. Other output types mostly come from separate models:
- Images: OpenAI's GPT-Image-2.5 Flare takes text and images and returns images, and Google's Nano Banana models generate and edit images.
- Video: Google's Gemini Omni Flash takes text, image and video input and generates video with native audio, 3 to 15 seconds per clip. Its current API does not accept audio input.
- Speech: Text-to-speech models such as Gemini 3.8 Flash TTS, plus the speech-to-speech models above.
How vision works in LLMs
Images are typically processed by a vision encoder (like ViT) that converts image patches into embeddings. These embeddings are projected into the LLM's token embedding space and concatenated with text tokens. The LLM then processes text and image tokens together using the same attention mechanism.
Limitations
Spatial reasoning (determining which object is to the left of another) remains a weak spot. Fine-grained OCR in complex layouts is unreliable. Most models process video by sampling frames rather than analyzing continuous motion, so events between sampled frames can be missed; Google's Live API, for example, takes camera images at up to 1 frame per second. Image resolution is another bottleneck: models often resize high-resolution images, losing fine details such as small text in screenshots.
Related terms
Models relevant to Multimodal
Gemini 4 Argon
Google's frontier model for real-world coding, enterprise knowledge work, and cybersecurity defense
View model →Gemini 3.8 Flash
Google's previous-generation Flash model for coding and agents, at half Gemini 4 Hydrogen's price until December 31, 2026
View model →Claude Opus 5.5
Anthropic's current Opus and its recommended starting model, priced at $4/$20 per 1M tokens
View model →GPT-6.1 Sol
Near-Astra intelligence for a fifth of the price
View model →Kimi K3.1 Pro
Moonshot AI's open-weight 1M-context flagship for long-horizon coding
View model →GPT-Live-1
A new generation of full-duplex voice models for natural human-AI conversation
View model →Gemini Omni Flash
Create anything from any input with conversational video editing
View model →