Hugo, my personal AI assistant.
A private, local-first AI assistant and command center I built for myself. It watches my projects, explains what changed, and answers by text or voice.
- In development · Private
- My Mac only, not deployed
- Text and real-time voice
Why it exists
Too many dashboards. One place to ask instead.
I run several web projects, each with its own dashboards: deployments, analytics, AI spend, subscriptions, generated media. Checking all of them every morning was slow, and I still missed things.
So I built one place that answers four questions: what is happening, what changed, what needs my attention, and what should I know. Fewer dashboards, one place to ask.
Grounded in tools
Every answer about my system comes from tools called in that turn. The model never sees the database, files, a shell or a credential.
Rules decide, the model explains
Project health and attention items are computed by deterministic rules with evidence. The assistant explains them and never recomputes them.
Asks before it acts
It observes and explains. The only changes it can make are tasks and subscriptions, and only after I confirm on screen or by saying yes.
Honest about gaps
Data a provider does not expose is shown as unavailable. Nothing is estimated or invented to fill a card.
How I communicate with it
Type or talk. Same tools, same conversation.
Text streams through the OpenAI Responses API. Voice is speech to speech over WebRTC: push-to-talk by default, and I can interrupt it mid-sentence. Try a replica below. It is scripted with fictional projects and never touches my real instance.
What it can do
Built and running on my Mac. Nothing here is a plan.
- Built
Command Center
A morning overview: system status, deterministic attention items with evidence, and a live feed of what changed.
- Built
Project monitoring
A project registry with read-only Vercel monitoring: deployments, domains, health, web analytics, usage and billing.
- Built
AI usage and costs
Reads usage and cost reporting from the OpenAI API when an admin key is configured, separate from the assistant's own key.
- Built
Text and voice assistant
Streamed text answers and speech-to-speech voice with push-to-talk, barge-in and a live transcript. Both share one tool registry.
- Built
Brain
A relationship graph of projects, providers, tools, models, skills and deployments, derived from Hugo's own data, with search, focus and path finding.
- Built
Studio
Image and video generation, social research and a montage editor. Paid runs are proposals that need my approval.
- Built
Tasks and subscriptions
Native tasks and manual subscription tracking. Assistant changes are confirm-before-execute and audited.
- Built
Skills registry
Lists Hugo's own capabilities and the AI skills installed on my machine, and flags new ones.
How it works inside
Rules compute. The model explains. Nothing acts without me.
Text
Streamed answers through the OpenAI Responses API, up to six tool rounds per turn.
Voice
Speech to speech over WebRTC with the OpenAI Realtime API. The server negotiates the session, the browser never sees a key.
One list for both
60 tools. Every call is validated with a strict schema. Outputs are trimmed and scrubbed of anything token-shaped.
Projects · health · activity
Deterministic rules with evidence.
Brain index
A graph derived in memory from Hugo's own data, never stored.
SQLite on my Mac
Deployments, activity, sync runs, tasks, Studio jobs and conversations. Raw audio is never stored.
Single flight, backoff
One central loop. A failed sync keeps the last good data and only degrades its own cards.
Read only
Vercel, OpenAI usage and costs, OpenRouter for Studio. Each declares what it can and cannot report.
The Brain
Hugo keeps a relationship graph of everything it knows about: projects, providers, tools, models, skills and deployments. It is rebuilt from Hugo's own data, so nothing in it is typed in by hand. This slice is fictional.
How it helps me
Less checking. More building.
- Open the Command Center: what changed overnight, what needs attention, nothing else.
- Ask instead of opening dashboards: did the deploy go through, which page is growing, what failed.
- Switch to voice. Same tools, same conversation, interrupt it any time.
- Generate images and video in Studio, with a cost shown before anything paid runs.
Deterministic first
Letting rules compute health and the model explain it made answers trustworthy. When the model guessed, I could not tell a real problem from a phrasing choice.
One registry, two interfaces
Text and voice call the same validated tools. Adding a capability once makes it work in both, and they never disagree.
Proposals, not actions
Every write is a card with Confirm and Cancel. It is slower by one click and the reason I trust it with anything at all.
Fewer moving parts in voice
A speech-to-text, model and text-to-speech chain was harder to interrupt and tune. Speech-to-speech over WebRTC replaced it in October.
Show unavailable as unavailable
A card that says Unavailable from API is more useful than a confident number that might be wrong.
What's next
Observe, recommend, then act, always with confirmation.
- PlannedGitHub: commits, pull requests and workflow runs (next phase)
- PlannedAI costs from more providers, with budgets and alerts
- PlannedA knowledge layer: notes, files and meeting decisions linked in the Brain
- PlannedA spoken morning briefing and a local wake word
- PlannedSocial metrics, and a daily briefing written over structured data
Development story
- App shell, Command Center, the Hugo Core visual, connector architecture, local SQLite.
- Project registry, read-only Vercel monitoring, central sync and health rules with evidence.
- Image and video generation moved in from a separate app, with approval for paid runs.
- Tasks and subscriptions, with the assistant's first confirm-before-execute actions.
- Text assistant and speech-to-speech voice on one tool registry.
- Brain v1 and the skills registry. Web analytics, usage and billing.
- In development · Private. Runs only on my Mac. GitHub is next.
Where this could go for a business
Potential applications of the same pattern. Hugo itself is personal: it is not offered as a product and none of these are built for clients.
An operations room for a small team
One screen for deployment health, SaaS spend and renewals across products, with the reasons behind every alert.
A read-only assistant over company data
Staff ask questions by text or voice; the assistant answers only through approved tools and asks before changing anything.
Client monitoring for an agency
Health and traffic for every client site in one place, so problems are found before the client reports them.
A map of tools and vendors
A relationship graph that shows which products depend on which providers, models and accounts.
Technical details
- Status
- In development · Private
- Runs on
- My Mac only, not deployed
- Interfaces
- Text and real-time voice
- Stack
- Next.js, React, TypeScript, SQLite, Zod
- AI
- OpenAI Responses API for text, OpenAI Realtime over WebRTC for voice, OpenRouter for Studio
- Assistant tools
- 60 in one registry shared by text and voice: 26 core, 23 for tasks and subscriptions (11 of them proposals), 11 for Vercel analytics and billing
- Storage
- SQLite through Drizzle, on the Mac
- Validation
- Zod schemas on every tool call
- Tests
- 30 Vitest files, more than 400 test cases
Start a project