Laurence Moroney - The AI Guy.

A Local Agent Loop with a Brain, a Voice, and Eyes

  • Oct 26, 2026

I love the moment when a stack of open models suddenly fits on one desk. Between October 22 and 24, three releases gave me a sketch I keep coming back to: a laptop or Jetson that chats with a small agent brain, speaks through a local voice model on Apple Silicon, and grounds camera crops with a perception model that draws boxes and masks.

A slim laptop and a small Jetson-style board on a warm desk, with glowing threads linking a speech wave, a camera crop, and a compact neural core

The brain: Nemotron 3.5 Bolt E4B

On October 23, NVIDIA released Nemotron 3.5 Bolt E4B, post-trained from Gemma-4-E4B-it for local voice and device assistants. NVIDIA describes it as an 8 billion parameter model with 4 billion effective parameters, under the OpenMDW-1.1 license, packing into about 4 GB as Q4_K_M. It ships conversational and thinking toggles, multi-turn tool use, and a 64K context window on NVIDIA’s own card.

The vendor quality claims, measured on NVIDIA’s harness, are roughly 40% higher average scores than Nemotron 3 Nano 4B, with up to about 80% fewer output tokens on its agent suites. That conciseness matters when you want speech-ready replies. The same day, Ollama v0.40.10 added nemotron-3.5-bolt:e4b. Ollama’s listing also presents Bolt as a vision-language model with a 128K context window, which is wider than the 64K text figure on NVIDIA’s page, so treat those packaging details as something to confirm on your own machine.

On Jetson hardware, NVIDIA’s Q4_K_M numbers show about 15.5 tokens per second on an Orin Nano 8GB for a short chat-shaped workload, which is enough to feel interactive when the replies stay short.

The voice: Phonon through Ollama

On October 24, Ollama 0.40.11 added speech models, starting with Gradium’s Phonon family on macOS. The path is simple:

ollama run phonon "Hello from Ollama."

You can also hit the OpenAI-compatible /v1/audio/speech endpoint, so existing TTS code mostly needs a new base URL. The English phonon model is Apache-2.0. The 40M multilingual variants are CC BY-NC. Ollama’s blog says its MLX path was about 35% faster than Gradium’s own runtime on a MacBook Pro with M5 Max, for 45 seconds of speech. Windows and Linux speech support are still coming, so for now this voice lives on Apple Silicon.

Piping a chat reply into Phonon is already a lovely demo: text starts speaking as the first sentence arrives. Pair that with Bolt’s conversational mode and you get a local loop that feels like talking to a small assistant rather than reading a wall of tokens.

The eyes: WeVisOne-VL

On October 22, Tencent released WeVisOne-VL, an Apache-2.0 perception model built on Qwen3-VL. The Hugging Face card lists the 4B variant at about 5 billion parameters, with a 2B sibling as well. The model is aimed at detection, OCR, counting, referring segmentation, and alpha matting, with a lightweight pixel decoder for masks and mattes.

I would use it as a specialist eye on camera crops: ask it to find the object, read the label, count the parts, or cut a clean matte, then pass structured results into Bolt as tool output. The authors are clear in their paper that limited general-purpose training data may weaken general chat relative to the Qwen3-VL base. That is fine for this sketch. Perception and packaging are the job here, and Bolt stays the conversational brain.

Optional pieces on the side

If you want a local coding specialist in the same week of releases, Recursive introduced Flint, an Apache-2.0 mixture-of-experts model with 40 billion total parameters and about 2.5 billion active per token. It is built for agentic coding on a laptop, and its total size still dominates disk, so plan the SSD accordingly.

For on-phone image generation packaging, iris-3b Turbo on LiteRT shows a full text-to-image path on a Galaxy S26, and the measured first image after install is on the order of minutes. Fun to know about, slow for a chat loop.

Keeping the sketch honest

A few limits I would keep in view while you wire this up:

  • Vendor harnesses. Bolt’s quality and token-savings numbers come from NVIDIA’s own suites. Ollama’s 128K and vision-language wording may not line up with NVIDIA’s 64K text card, so check the build you actually pull.
  • Specialist roles. WeVisOne is a perception model. Expect it to shine on grounding tasks and possibly lag the Qwen3-VL base on open chat.
  • Platforms. Phonon speech in Ollama is macOS-only for now. Jetson and Windows boxes can still run Bolt and WeVisOne while you wait for speech elsewhere, or stream audio from a Mac on the same desk.
  • Disk. Flint’s roughly 40 billion total parameters are a serious download next to Bolt’s 4 GB target.

What I would build first

Here is the weekend experiment I would try. Put Bolt on a Jetson or a laptop GPU as the agent that owns tools and conversation. On a Mac beside it, run Phonon and pipe Bolt’s short replies into speech. Point a webcam at a desk, crop interesting frames, and send those crops to WeVisOne-VL for boxes, OCR, counts, or mattes. Feed the structured results back to Bolt as tool messages so the spoken reply can name what it saw.

None of this needs a cloud API key. It does need curiosity, a bit of disk, and a willingness to treat each model as a specialist rather than as one giant do-everything brain. If you build a version of this loop, I would love for you to share what worked, especially the glue code around tool calls and camera crops. That glue is where local agents become real.

Read the sources

Search