Vision & capabilities

Northern Inference passes images straight through to vision-capable models (Gemini, Claude, Gemini 2.5 Pro-class, and others). Two things are worth knowing: how to send an image, and how a capability-aware client can detect that a route supports images.

How NI advertises capabilities

Every model route NI serves carries capability metadata on both /v1/models and /api/billing/models. The OpenAI-style /v1/models response adds NI fields on each model object:

curl -sS https://northerninference.ca/v1/models \
  -H "Authorization: Bearer $NI_API_KEY" | jq '.data[] | {id, supports_vision, capabilities}'

You will see, per route:

{
  "id": "vertex_ai/gemini-2.5-pro-ca",
  "supports_vision": true,
  "capabilities": {
    "supports_vision": true,
    "modalities": ["text", "image"],
    "context_window": 1048576,
    "max_output_tokens": 65535,
    "supports_function_calling": true
  }
}

supports_vision is mirrored at the top level so a client that reads only flat fields can still detect it. The same capabilities object is on each entry of /api/billing/models.

These facts are data-driven: NI resolves them from the model capability store (the upstream model catalog), then a curated base-model family match, then a conservative default. They are not a hand-maintained list, so newly discovered models carry capabilities automatically.

Send an image (OpenAI image_url content block)

Send the image as a content block on the user message. A base64 data URL works without any upload step:

B64=$(base64 -i screenshot.png | tr -d '\n')
curl -sS https://northerninference.ca/v1/chat/completions \
  -H "Authorization: Bearer $NI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "vertex_ai/gemini-2.5-pro-ca",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "What does this image say? Quote the text exactly."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,'"$B64"'"}}
      ]}
    ],
    "max_tokens": 300
  }'

A public https:// image URL works too (the provider fetches it):

{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}

Pick any route whose supports_vision is true. Gemini 2.5 pro/flash, Claude 4.x, and Gemini 2.5 Pro-class routes all qualify.

Anthropic-native /v1/messages

If you call /v1/messages instead of /v1/chat/completions, use the Anthropic image block shape:

{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "<BASE64>"}}

If your client refuses to attach images

Some clients (Cursor, Continue, and other capability-aware tools) keep a built-in table of which model names support images, keyed on standard names like gpt-4o or gemini-2.5-pro. NI route IDs are custom (for example vertex_ai/gemini-2.5-pro-ca), so a client with a hardcoded table will not match the route and will refuse to attach the image, stripping it before it ever reaches NI. The request then arrives as text only and the model never sees the picture.

Two knobs fix this:

1. Use a client that reads capabilities from /v1/models. It will pick up

supports_vision and allow images on NI routes automatically.

1. For a client with a hardcoded table, set a per-model override so the

client treats the NI route as vision-capable. Most such clients expose a per-model "supports images" / "vision" toggle in their model settings. Turn it on for the NI route you added. (See the client-specific guide, for example cursor.md, for where that toggle lives.)

If images still do not arrive, confirm the client is actually embedding the image as an image_url block and not sending a file path as plain text. You can verify NI received an image by checking the request in portal → Usage: a vision request shows image content parts.