What is multimodal AI?

What it means at work

In practice it means you can attach a photo, screenshot, chart, scanned form or recording and ask about it in words. It saves the step of retyping. Common uses include reading a chart in a slide, pulling fields from a scanned invoice, describing a whiteboard photo, or turning a recorded call into notes.

What the documentation says

• Images in chat. The Anthropic vision guide says you can upload an image like you would a file or drag and drop it into the chat window in claude.ai.
• Whole documents. The Google Gemini API documentation says Gemini models can process documents in PDF format, and handle long documents up to 1000 pages.
• What counts as multimodal. The GPT-4 Technical Report describes a multimodal model which can accept image and text inputs and produce text outputs.
• Examples of capability. A Survey on Multimodal Large Language Models cites capabilities such as writing stories based on images.
• Known weak spots. The Anthropic vision guide says Claude might hallucinate or make mistakes when interpreting low-quality images. The same guide names very small images under 200 pixels as another weak case.
• Counting. The Anthropic vision guide says Claude can give approximate counts of objects in an image but might not always be precisely accurate.

An example

After a workshop, you photograph the whiteboard and ask, ‘Turn this into a list of decisions and open questions with owners.’ The assistant reads the handwriting and produces the list. You correct the two names it misread.

A common misconception

It is easy to assume that if a tool can look at an image, it sees it perfectly. Small text, poor lighting and dense tables cause errors. On this, the Anthropic guide says image interpretations need careful review especially for high-stakes use cases. Treat extracted numbers and names as needing a quick check against the original. It also matters what you are allowed to upload. A photo of a whiteboard or a scanned contract may contain client details, so the same data rules apply as for typed text.

Sources

• Anthropic, Vision: “Upload an image like you would a file … drag and drop an image directly into the chat window”
• Google AI for Developers, Document understanding (Gemini API): “Gemini models can process documents in PDF format”
• Google AI for Developers, Document understanding (Gemini API): “long documents up to 1000 pages”
• OpenAI, “GPT-4 Technical Report” (arXiv): “multimodal model which can accept image and text inputs and produce text outputs”
• “A Survey on Multimodal Large Language Models” (arXiv): “such as writing stories based on images”
• Anthropic, Vision: “Claude might hallucinate or make mistakes when interpreting low-quality”
• Anthropic, Vision: “or very small images under 200 pixels”
• Anthropic, Vision: “Claude can give approximate counts of objects in an image but might not always be precisely accurate”
• Anthropic, Vision: “especially for high-stakes use cases”