AI/ML,Full-Stack,Voice
Voice Agent Builder
Build a voice agent from your own documents, then call it on a real phone number. Runs on a managed platform or a self-hosted media pipeline, with cost and latency measured both ways.
Upload documents, get a phone number that answers questions about them. The same agent runs on two completely different phone stacks behind one interface, so switching vendors is a single environment variable.
- Grounded in your own files. Uploads are chunked, embedded and stored in SQLite. Every reply is retrieved from those chunks, so the agent quotes your pricing rather than inventing it.
- No vector database. Embeddings are Float32Arrays in SQLite and cosine similarity runs in memory. At a few hundred chunks per agent that is microseconds, and it removes a service from the diagram.
- Two interchangeable phone stacks. A managed platform (Vapi) and a self-hosted pipeline (Twilio Media Streams, Deepgram, OpenAI, Deepgram Aura) sit behind one provider interface. The retrieval logic never learns which vendor is live.
- Zero audio transcoding. Twilio speaks mu-law 8kHz, Deepgram accepts it, Aura emits it. Bytes cross the entire pipeline without a resample.
- Built for interruption. Synthesis starts at the first sentence boundary, before the model has finished writing, and Twilio mark events drive barge-in when the caller talks over the agent.
How a call flows
Inbound to a Twilio number, or outbound from the agent page. The caller never learns which stack answered.
one environment variable
Both stacks implement the same provider interface, so swapping vendors changes a string and nothing else. The retrieval and prompting code never learns which one is live.
$0.090 / min · 2,346ms per turn
Vapi owns telephony, transcription, turn-taking and voice. The app exposes an OpenAI-compatible endpoint and Vapi calls it once per turn for the reply.
$0.026 / min · ~1,150ms per turn
A Node WebSocket server bridges Twilio Media Streams to Deepgram for transcription and Aura for voice. Turn detection, barge-in and buffering are all mine to get right.
The question is embedded and compared against the agent's chunks by cosine similarity, and the top three are pasted into the system prompt. Embeddings are Float32Arrays in SQLite and the comparison runs in memory: no vector database, microseconds at this scale.
1.7% of the bill
Tokens stream back, and synthesis begins at the first sentence boundary rather than waiting for the full reply. The model is the cheapest part of the call by an order of magnitude.
mu-law 8kHz end to end
Twilio speaks mu-law 8kHz, Deepgram accepts it, and Aura emits it. Bytes cross the whole pipeline without a resample, which is why both halves are Deepgram rather than the best of each vendor.
Twilio mark events track what has actually played, so when the caller talks over the agent the queued audio is cleared instead of finishing the sentence into a conversation that has moved on.
Measured on real calls
| Managed (Vapi) | Self-hosted | |
|---|---|---|
| Platform fee | $0.0503 | none |
| Voice | $0.0285 | $0.0080 |
| Transcription | $0.0100 | $0.0077 |
| Telephony | included | $0.0085 |
| Language model | $0.0015 | $0.0015 |
| Cost per minute | $0.0903 | $0.0257 |
| Turn latency | 2,346ms | ~1,150ms |
The language model is 1.7% of the bill and the platform fee is 56%, which inverts the obvious optimisation. Trimming the system prompt beat every model change on latency, because it is re-sent every turn and prefill dominated. Self-hosting only breaks even above roughly 600 minutes a month, once the fixed server cost is counted.
Future work
- Confirm barge-in against real speech. It works against synthetic audio, 401ms from interruption to flushing Twilio's buffer, but the thresholds are environment variables because they are expected to need tuning on real callers.
- Warm transfer to a human, so an agent can hand off mid-call rather than dead-ending.
- Voicemail detection and call recording playback, both table stakes for anything running unattended.
- Consent and do-not-call management. Automated outbound calling is regulated: the FCC treats AI-generated voices as artificial under the TCPA, and several states require disclosing that the caller is an AI. That is a compliance problem before it is a technical one.