Frostphone
AI telephony that runs entirely on the customer’s own hardware
A voice agent that answers the phone for doctors’ practices, law firms and tax advisors, running entirely on hardware in their own building.
01The problem
A voice agent in somebody else’s cloud is a solved problem. A voice agent for a medical practice is not. The caller says why they need an appointment, and that sentence is health data. Send it to a third-party API and the practice has to justify that transfer - which, for health data, most of them cannot.
So the product has no cloud version. The whole pipeline is installed on the customer’s own machine, and the call never reaches a network they do not control. That is the only version of this product a practice can sign off on, so it is the only one that exists.
02What I built
A Docker stack of twelve services on the customer’s own machine - ten in a development setup, plus the reverse proxy and the SIP gateway that a production install turns on. The ones that matter: LiveKit for real-time media, LiveKit SIP so the agent answers an actual phone number, Whisper for transcription, neural speech synthesis, Ollama for the language model, and the agent process that orchestrates the three. Around them a configuration dashboard, n8n for the automations a customer wires in afterwards, and monitoring.
Language, voice and model are all configuration rather than build-time decisions. Whichever speech and synthesis models the customer loads is what the agent speaks; adding another language means adding another model, not another release.
Tools let the agent reach out mid-call through webhooks and n8n - booking an appointment, querying a record, writing into a CRM - so it can finish a conversation rather than take a message.
03Tuning how it listens
Most of what makes a phone agent feel human is turn-taking, and almost none of it is the language model. The dashboard exposes the part that actually decides whether a caller feels heard or trampled on: how long the agent waits after silence before it answers (0.3 to 2 seconds), how sensitive voice activity detection is, how much silence has to pass before it triggers, and how many words a caller needs before they are allowed to interrupt.
It also knows the difference between an interruption and agreement. "Mhm", "okay" and "ja" are the sounds people make to show they are still listening, and treating them as an interruption makes an agent that stops mid-sentence every few seconds. Filler words can be ignored, and a live indicator shows what the detector is hearing while a call is running.
On the model side: response length, temperature and top-p, with defaults set for speech, not for chat - lower temperature and short answers, because a voice agent that gets creative is a voice agent that rambles at somebody holding a phone.
04What the operator sees
Every call is logged with its transcript, duration, the voice used and optionally an MP3 - searchable, with statistics per day, week, month or year, and a retention period after which a background job deletes transcripts and audio whether or not anybody opens the dashboard.
A system tab shows CPU, RAM, disk and GPU load with VRAM and temperature, and every container with its uptime, resource use, health check and response time. Each can be started, stopped and restarted from the browser, plus one button that frees the GPU by stopping the three inference services at once - for the customer whose card is also doing something else.
The control surface names its own risks. Stopping a container by hand keeps it stopped across a reboot, so a "just for a moment" voice agent is still refusing calls weeks later, and the confirmation dialog says so. Stopping the frontend or the proxy removes the interface you would use to bring them back. And because the route reaches the host’s Docker daemon, the admin password is a prerequisite rather than a recommendation - customers who do not want that capability can remove the socket, and the rest of the dashboard keeps working without it.
05Decisions worth explaining
Reasoning models are the wrong tool for a phone call
Models that think before answering produce their reasoning tokens before the first spoken word, and the caller hears every one of them as silence. The default is an 8B model, picked for German and function calling instead of benchmark position, because on a phone line the latency budget is the specification.
A finished installation is not a release
Nothing here deploys centrally. Every installation is prepared for one named customer, and the rollout is gated behind fifty-seven written checks in eight groups - scope, roles and contracts, special-category data, caller consent, processors and transfers, security evidence, deletion and operations, and sign-off. The whole thing stays blocked while any one of them is unanswered, and “not applicable” needs a reason and a name against it. Installing the software successfully is explicitly not the same as being allowed to run it, because for a practice handling health data those are two different questions and only one of them is technical.
The hardware decides the model, not the other way around
Measured on a 12 GB card: the language model takes 6.0 GB, speech recognition 1.3 GB and synthesis 2.4 GB. A bigger card simply means a stronger model, and more cards lift the ceiling again. Six models are vetted and carried in the dashboard with their measured VRAM cost and a note on what each is for, and the list is filtered against what is actually free on that machine instead of against a fixed assumption - which turns a support ticket reading "it stopped responding" into a choice the customer cannot make wrongly.
Keeping the CUDA context after unloading the model
Idle unload frees roughly 9 GB but deliberately leaves about 200 MB of CUDA context per service standing. That is the whole reason a reload takes one second for speech recognition and three for synthesis instead of starting cold. The language model still costs about twenty seconds, so the first call after a quiet night is slow and everything after it is not - and on a dedicated machine the models simply stay resident, because the alternative is making a caller wait for a decision the operator already made.
Questions about this project?
I am happy to walk through the architecture, the parts that did not work, or the code itself.