Vision & capabilities

Northern Inference passes images straight through to vision-capable models (Gemini, Claude, Gemini 2.5 Pro-class, and others). Two things are worth knowing: how to send an image, and how a capability-aware client can detect that a route supports images.

How NI advertises capabilities

Every model route NI serves carries capability metadata on both /v1/models and /api/billing/models. The OpenAI-style /v1/models response adds NI fields on each model object:

curl -sS https://northerninference.ca/v1/models \
  -H "Authorization: Bearer $NI_API_KEY" | jq '.data[] | {id, supports_vision, capabilities}'

You will see, per route:

{
  "id": "claude-sonnet-4-6-t4",
  "supports_vision": true,
  "capabilities": {
    "supports_vision": true,
    "modalities": ["text", "image"],
    "context_window": 1048576,
    "max_output_tokens": 65535,
    "supports_function_calling": true
  }
}

supports_vision is mirrored at the top level so a client that reads only flat fields can still detect it. The same capabilities object is on each entry of /api/billing/models.

These facts are data-driven: NI resolves them from the model capability store (the upstream model catalog), then a curated base-model family match, then a conservative default. They are not a hand-maintained list, so newly discovered models carry capabilities automatically.

Send an image (OpenAI image_url content block)

Send the image as a content block on the user message. A base64 data URL works without any upload step:

B64=$(base64 -i screenshot.png | tr -d '\n')
curl -sS https://northerninference.ca/v1/chat/completions \
  -H "Authorization: Bearer $NI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "claude-sonnet-4-6-t4",
    "messages": [
      {"role": "user", "content": [
        {"type": "text", "text": "What does this image say? Quote the text exactly."},
        {"type": "image_url", "image_url": {"url": "data:image/png;base64,'"$B64"'"}}
      ]}
    ],
    "max_tokens": 300
  }'

A public https:// image URL works too (the provider fetches it):

{"type": "image_url", "image_url": {"url": "https://example.com/photo.jpg"}}

Pick any route whose supports_vision is true. Gemini 2.5 pro/flash, Claude 4.x, and Gemini 2.5 Pro-class routes all qualify.

Anthropic-native /v1/messages

If you call /v1/messages instead of /v1/chat/completions, use the Anthropic image block shape:

{"type": "image", "source": {"type": "base64", "media_type": "image/png", "data": "<BASE64>"}}

If your client refuses to attach images

Some clients (Cursor, Continue, and other capability-aware tools) keep a built-in table of which model names support images, keyed on standard names like gpt-4o or gemini-2.5-pro. NI route IDs are custom (for example claude-sonnet-4-6-t4), so a client with a hardcoded table will not match the route and will refuse to attach the image, stripping it before it ever reaches NI. The request then arrives as text only and the model never sees the picture.

Two knobs fix this:

1. Use a client that reads capabilities from /v1/models. It will pick up

supports_vision and allow images on NI routes automatically.

1. For a client with a hardcoded table, set a per-model override so the

client treats the NI route as vision-capable. Most such clients expose a per-model "supports images" / "vision" toggle in their model settings. Turn it on for the NI route you added. (See the client-specific guide, for example cursor.md, for where that toggle lives.)

If images still do not arrive, confirm the client is actually embedding the image as an image_url block and not sending a file path as plain text. You can verify NI received an image by checking the request in portal → Usage: a vision request shows image content parts.

Reasoning and thinking effort

Models that think before answering (Claude extended thinking, OpenAI o-series and gpt-5+ reasoning) expose two different controls. Pick the portable one unless you are certain your route is Anthropic.

Use reasoning_effort for cross-provider portability

reasoning_effort takes low, medium, or high and is the portable control:

• On an OpenAI reasoning route (o-series, gpt-5+) it maps to that model's native reasoning effort.
• On a Claude route, NI maps it to an Anthropic thinking budget for you (low about 8k, medium about 25k, high about 64k thinking tokens).

{
  "model": "claude-sonnet-4-6-t4",
  "messages": [{"role": "user", "content": "Plan a refactor of this module."}],
  "reasoning_effort": "high",
  "max_tokens": 70000
}

thinking budget_tokens is Anthropic-native, auto-translated elsewhere

The Anthropic-native thinking block (and Claude Code's effort flag, which sends it) is native on Anthropic routes. On a non-Anthropic route, NI automatically converts the thinking budget into the portable reasoning_effort (bucketed to low, medium, or high) so your reasoning request still runs on GPT, Gemini, DeepSeek, and other reasoning models, instead of being dropped. That conversion is coarse (an exact token budget becomes a three-level effort), so for precise cross-provider control prefer reasoning_effort directly. A route with no reasoning mode at all (for example Llama or Mistral) returns a normal answer with no reasoning.

max_tokens must exceed the thinking budget

The thinking budget is spent out of max_tokens. If your budget is 32000, max_tokens must be larger (for example 40000), or the request returns a 400 because there is no room left for the visible answer. Rule of thumb: set max_tokens to at least the thinking budget plus the longest answer you expect.

Where the reasoning text goes

• Anthropic routes return the thinking text in a thinking content block (or reasoning_content on the OpenAI-compatible path), so you can show or log it.
• OpenAI o-series routes hide the raw reasoning; you get only the final answer plus a reasoning-token count in usage.

NI bills thinking tokens explicitly using each route's thinking rate, with no double-billing against output tokens.