Vision
Send images to multimodal models with content parts — a Pro feature. URL vs. inline base64, the Gemini URL caveat, and choosing a vision-capable model.
Pro Vision lets a model read images alongside text. In Radon you
supply images as content parts on a user message, and the same shape works
on every vision-capable provider.
Vision is a Pro feature on any provider — set a licenseKey and call
await ai.init() first. See Free vs. Pro.
await ai.chat({
provider: "openai",
messages: [
{
role: "user",
content: [
{ type: "text", text: "What's in this image?" },
{ type: "image", source: { url: "https://example.com/cat.png" } },
],
},
],
});Content parts
When content is an array instead of a string, each element is a content part
— either text or an image. Mix as many as you like in one message.
text{ type: 'text'; text: string }A run of text.
image{ type: 'image'; source: ImageSource }An image, sourced by URL or inline base64 (see below).
URL vs. inline base64
An image source is either a URL the provider fetches, or inline base64 bytes
with a MIME type.
// A URL the provider fetches:
{ type: "image", source: { url: "https://example.com/cat.png" } }
// Inline base64 bytes:
{ type: "image", source: { data: "<base64>", mimeType: "image/png" } }Gemini can't fetch arbitrary web URLs
OpenAI and Anthropic accept image URLs. Gemini cannot fetch arbitrary web
URLs — Radon passes a URL as a Gemini fileData.fileUri, which expects a File
API / GCS URI, not a public web link. For Gemini, prefer inline base64
(source: { data, mimeType }). Radon does not download-and-reupload images on your
behalf.
Choosing a vision-capable model
capabilities.vision is a provider-can-ever flag — whether a given request
works also depends on the model you pick. A text-only model on a
vision-capable provider will reject images. Choose a multimodal model (e.g. an
OpenAI gpt-4o family model, a Mistral Pixtral model, a Grok vision model).
| Provider | Vision |
|---|---|
openai | ✅ |
anthropic | ✅ |
groq | ✅ (select models) |
google | ✅ |
mistral | ✅ (Pixtral) |
openrouter | ✅ (routed-model dependent) |
xai | ✅ (Grok vision) |
together | ✅ (select models) |
ollama | ✅ (e.g. llava) |
deepseek | ❌ |
Base64 works everywhere
When in doubt — or when portability across providers matters — send inline base64. It's the one image form every vision-capable provider accepts.
Reading a local file as base64
import { readFile } from "node:fs/promises";
const bytes = await readFile("./cat.png");
const data = bytes.toString("base64");
await ai.chat({
provider: "google",
messages: [
{
role: "user",
content: [
{ type: "text", text: "Describe this photo." },
{ type: "image", source: { data, mimeType: "image/png" } },
],
},
],
});Other media isn't in the unified shape yet
Audio, video, and document/PDF input aren't part of the unified content shape.
Reach them through providerOptions or native().
Next steps
Embeddings
Turn text into vectors with ai.embed() — a Pro feature. Which providers support it, batching, dimensions, and the normalized EmbedResult.
Structured output
Ask for JSON or a schema-conforming object with responseFormat — a Pro feature. JSON mode, JSON Schema, what each provider guarantees, and the Anthropic caveat.