clarilotENGINEERING NOTE · SEPTEMBER 25, 2026
Why put a host between a voice endpoint and an agent?
Keep the pocket device responsible for a clear press-to-speak interaction, bounded audio capture and playback. Put transcription, agent routing, credentials, permission checks and durable work on an independently operated host. That separation makes failures and privileges easier to reason about; it does not, by itself, guarantee low latency, privacy or safe agent behavior.
What crosses each boundary?
Person → device: a held button starts one utterance; releasing it ends capture. A local acknowledgement can confirm the press without pretending that the agent has finished. Hold-to-talk is a deliberate first-step UX choice, not a claim that wake-word detection or far-field audio works.
Device → host: a bounded audio frame stream and an utterance identifier cross an authenticated transport. A production host should reject invalid sizes and replayed turns; the device should not contain the host agent's credentials. Authentication of a device is not authorization for every action its transcript might describe.
Host → agent: speech is transcribed, then an explicit routing policy chooses the permitted workflow. A transcript can contain mistakes or adversarial instructions, so the gateway must treat it as input rather than authority to grant new tools. OWASP's prompt-injection guidance recommends least-privilege tool scopes and validating tool calls against the user's permissions and session context. A spoken request for a sensitive action still needs the host-side controls appropriate to that action.
Host → device: a job result becomes reply audio; an acknowledgement should represent playback, not merely a successful network send. If a connection drops, the host should retain a claimable result until playback is confirmed, while the endpoint should distinguish “accepted,” “working,” “ready to play,” and “played.” These are design states, not measured Clarilot response-time figures.
A failure example: the connection drops after the answer is prepared
Imagine a short request is accepted, but Wi-Fi disconnects before the spoken answer reaches the speaker. If the gateway marks the answer complete when it sends audio, the user may hear nothing and the result is lost. A recoverable design leases the result, lets the device reconnect and claim it again, and acknowledges only completed playback. The host also needs an idempotent utterance or job key so a reconnect does not run the same action twice. A replay is not a substitute for confirming what the user actually heard.
What this project can and cannot establish today
Clarilot's current host prototype implements an authenticated audio WebSocket, a host-side speech/agent workflow and result-lease handling; the firmware has compiled. This is a project-specific integration, not a general adapter for arbitrary self-hosted agents. Physical microphone, speaker and battery performance remain unvalidated; there is no measured end-to-end pocket-device latency or field reliability claim. The public site is a preview and waitlist, not a gateway endpoint.
For comparison, Home Assistant's Assist pipeline documentation separates wake word, speech-to-text, intent and text-to-speech stages. That is another project's published architecture, not a Clarilot integration or performance benchmark. Clarilot's first target is the smaller push-to-talk interaction; where to place a wake word, if ever, depends on real power, noise and usability measurements.
What should an engineering review measure next?
- Record button-to-local-feedback and release-to-first-audible-reply separately, including p50/p95 on the intended network.
- Test silent speech, noise, interrupted Wi-Fi, reconnect and duplicate utterance IDs; confirm no repeated agent action or lost played/unplayed result.
- Measure microphone pickup and speaker intelligibility on the actual device, plus battery behavior and high-volume power stability.
- Audit which secret lives on each side of the device/host boundary and which workflow each device may request.
Want project updates as the physical tests happen? Join the Clarilot email waitlist. It is not a purchase, beta access guarantee or device reservation.