In development · Private · AI agents · Automation · Experimental apps

Hugo, my personal AI assistant.

A private, local-first AI assistant and command center I built for myself. It watches my projects, explains what changed, and answers by text or voice.

Status
In development · Private
Runs on
My Mac only, not deployed
Interfaces
Text and real-time voice

Not a product and not public. Everything you can click on this page runs on fictional data.

Why it exists

Too many dashboards. One place to ask instead.

I run several web projects, each with its own dashboards: deployments, analytics, AI spend, subscriptions, generated media. Checking all of them every morning was slow, and I still missed things.

So I built one place that answers four questions: what is happening, what changed, what needs my attention, and what should I know. Fewer dashboards, one place to ask.

How it differs from a chatbot

  • Grounded in tools

    Every answer about my system comes from tools called in that turn. The model never sees the database, files, a shell or a credential.

  • Rules decide, the model explains

    Project health and attention items are computed by deterministic rules with evidence. The assistant explains them and never recomputes them.

  • Asks before it acts

    It observes and explains. The only changes it can make are tasks and subscriptions, and only after I confirm on screen or by saying yes.

  • Honest about gaps

    Data a provider does not expose is shown as unavailable. Nothing is estimated or invented to fill a card.

How I communicate with it

Type or talk. Same tools, same conversation.

Text streams through the OpenAI Responses API. Voice is speech to speech over WebRTC: push-to-talk by default, and I can interrupt it mid-sentence. Try a replica below. It is scripted with fictional projects and never touches my real instance.

Loading the interactive demo…

What it can do

Built and running on my Mac. Nothing here is a plan.

  • Built

    Command Center

    A morning overview: system status, deterministic attention items with evidence, and a live feed of what changed.

  • Built

    Project monitoring

    A project registry with read-only Vercel monitoring: deployments, domains, health, web analytics, usage and billing.

  • Built

    AI usage and costs

    Reads usage and cost reporting from the OpenAI API when an admin key is configured, separate from the assistant's own key.

  • Built

    Text and voice assistant

    Streamed text answers and speech-to-speech voice with push-to-talk, barge-in and a live transcript. Both share one tool registry.

  • Built

    Brain

    A relationship graph of projects, providers, tools, models, skills and deployments, derived from Hugo's own data, with search, focus and path finding.

  • Built

    Studio

    Image and video generation, social research and a montage editor. Paid runs are proposals that need my approval.

  • Built

    Tasks and subscriptions

    Native tasks and manual subscription tracking. Assistant changes are confirm-before-execute and audited.

  • Built

    Skills registry

    Lists Hugo's own capabilities and the AI skills installed on my machine, and flags new ones.

Recreated in HTML with fictional projects. Not a screenshot: the real interface shows private projects, accounts and costs, so it is not published.

How it works inside

Rules compute. The model explains. Nothing acts without me.

Youtype or talk
  1. 01 · Interfaces

    Text

    Streamed answers through the OpenAI Responses API, up to six tool rounds per turn.

    Voice

    Speech to speech over WebRTC with the OpenAI Realtime API. The server negotiates the session, the browser never sees a key.

  2. 02 · Tool registry

    One list for both

    60 tools. Every call is validated with a strict schema. Outputs are trimmed and scrubbed of anything token-shaped.

  3. 03 · Query layer

    Projects · health · activity

    Deterministic rules with evidence.

    Brain index

    A graph derived in memory from Hugo's own data, never stored.

  4. 04 · Local storage

    SQLite on my Mac

    Deployments, activity, sync runs, tasks, Studio jobs and conversations. Raw audio is never stored.

  5. 05 · Sync

    Single flight, backoff

    One central loop. A failed sync keeps the last good data and only degrades its own cards.

  6. 06 · Connectors

    Read only

    Vercel, OpenAI usage and costs, OpenRouter for Studio. Each declares what it can and cannot report.

Simplified and sanitized. Requests flow down, data flows up; the model only ever sees validated tool output.

The Brain

Hugo keeps a relationship graph of everything it knows about: projects, providers, tools, models, skills and deployments. It is rebuilt from Hugo's own data, so nothing in it is typed in by hand. This slice is fictional.

Loading the interactive graph…

How it helps me

Less checking. More building.

  1. Start of dayOpen the Command Center: what changed overnight, what needs attention, nothing else.
  2. While buildingAsk instead of opening dashboards: did the deploy go through, which page is growing, what failed.
  3. Hands busySwitch to voice. Same tools, same conversation, interrupt it any time.
  4. Making contentGenerate images and video in Studio, with a cost shown before anything paid runs.

What building it taught me

  • Deterministic first

    Letting rules compute health and the model explain it made answers trustworthy. When the model guessed, I could not tell a real problem from a phrasing choice.

  • One registry, two interfaces

    Text and voice call the same validated tools. Adding a capability once makes it work in both, and they never disagree.

  • Proposals, not actions

    Every write is a card with Confirm and Cancel. It is slower by one click and the reason I trust it with anything at all.

  • Fewer moving parts in voice

    A speech-to-text, model and text-to-speech chain was harder to interrupt and tune. Speech-to-speech over WebRTC replaced it in October.

  • Show unavailable as unavailable

    A card that says Unavailable from API is more useful than a confident number that might be wrong.

What's next

Observe, recommend, then act, always with confirmation.

Planned, not built

  • PlannedGitHub: commits, pull requests and workflow runs (next phase)
  • PlannedAI costs from more providers, with budgets and alerts
  • PlannedA knowledge layer: notes, files and meeting decisions linked in the Brain
  • PlannedA spoken morning briefing and a local wake word
  • PlannedSocial metrics, and a daily briefing written over structured data

Development story

Stages in the order of the project roadmap.

  1. FoundationApp shell, Command Center, the Hugo Core visual, connector architecture, local SQLite.
  2. ProjectsProject registry, read-only Vercel monitoring, central sync and health rules with evidence.
  3. StudioImage and video generation moved in from a separate app, with approval for paid runs.
  4. OperationsTasks and subscriptions, with the assistant's first confirm-before-execute actions.
  5. AssistantText assistant and speech-to-speech voice on one tool registry.
  6. BrainBrain v1 and the skills registry. Web analytics, usage and billing.
  7. NowIn development · Private. Runs only on my Mac. GitHub is next.

Where this could go for a business

Potential applications of the same pattern. Hugo itself is personal: it is not offered as a product and none of these are built for clients.

  • An operations room for a small team

    One screen for deployment health, SaaS spend and renewals across products, with the reasons behind every alert.

  • A read-only assistant over company data

    Staff ask questions by text or voice; the assistant answers only through approved tools and asks before changing anything.

  • Client monitoring for an agency

    Health and traffic for every client site in one place, so problems are found before the client reports them.

  • A map of tools and vendors

    A relationship graph that shows which products depend on which providers, models and accounts.

Talk about a system like this

Technical details

Status
In development · Private
Runs on
My Mac only, not deployed
Interfaces
Text and real-time voice
Stack
Next.js, React, TypeScript, SQLite, Zod
AI
OpenAI Responses API for text, OpenAI Realtime over WebRTC for voice, OpenRouter for Studio
Assistant tools
60 in one registry shared by text and voice: 26 core, 23 for tasks and subscriptions (11 of them proposals), 11 for Vercel analytics and billing
Storage
SQLite through Drizzle, on the Mac
Validation
Zod schemas on every tool call
Tests
30 Vitest files, more than 400 test cases