I wanted to build a radio phone-in show where a human host could talk to fictional AI callers, which sounds quite straightforward until you remember that a radio phone-in is not simply two voices making noises at one another, because it also has a running order, a producer, callers waiting to come on, private notes, soundchecks, interruption, timing, graphics, audio levels, awkward silences and natural-born chaos.
Github: https://github.com/paolodit/phone-in-studio
The first version of the idea was not a specification, it was more like a pile of product thoughts that had been shower thoughts combined with coffee-talk and girlfriend-pretending-to-be-interested thoughts.
….watch the demo, it makes much more sense!
I wanted passably-believable callers with Jerry Springer issues, different textured and vocabulary voices and different reasons for “calling”, I wanted to be able to test them privately, I wanted a host to bring them on air one at a time, I wanted a clean visual output for OBS and TikTok Live Studio, and I wanted the whole thing to feel like a piece of broadcasting software rather than an API demonstration – I wanted it real!
Codex was pretty good at turning individual pieces of that my thoughts into software, but the interesting part was not that it could produce React components, Prisma models or API routes quickly. The interesting part was learning how to build a harness around Codex so that its work kept moving towards the same product, and then building another harness inside the product so that unpredictable voice models could be used by somebody who was trying to host a show rather than conduct a live seminar on WebRTC.
That I-want-it-to-be-real distinction became the whole project, the model was capable, the APIs were capable, and Codex was capable, but capability was not the scarce ingredient; The scarce ingredient was taste, by which I mean deciding what the software should feel like, what the host should be thinking about at any particular moment, what should happen automatically, what should require a deliberate click, and what should be removed even though it had taken an annoyingly impressive amount of work to build.
I did not start with a perfect brief
I use Codex most effectively when I start by telling it what I am actually trying to achieve, including the scrappy bits, the contradictions and the things I have not decided yet, rather than pretending I have arrived with a fully formed technical specification delivered on a stone tablet. Codex has become really good at creating coherence from unstructured chit-chat.
For this project, the important sentence was not “build an AI voice application.” It was “I want to host a radio phone-in.” – this changes the shape of the work, because a host does not want to manage session tokens, inspect JSON, choose a voice activity detection threshold or wonder whether an audio buffer has been cleared – The host wants to see who is calling, understand why they are calling, answer them, interrupt them, put them on hold, bring in the next person and keep the programme moving, for the next insane Jerry Springer style guest.
The conversations enabling framework that this system produces have to be a durable structure: Caller, candidate, approved caller, queue item, incoming, connecting, live, held and ended became real product concepts. Private production notes and public broadcast information became separate data boundaries.
So, to empower this, voice providers (OpenAI, ElevenLabs, Fish, Gemini) became replaceable implementations behind one interface.
And the goal was to have a show lifecycle that became a state machine – rather than a collection of buttons.
The harness gradually became
The show state machine knows that a caller cannot be answered before they are incoming, that a caller cannot be put on hold before they are live, and that the next caller cannot be cued when there is nobody left in the queue – Obvious stuff to a human, but not so obvious to codey!
The broadcast serializer uses an allow-list, so a caller’s public name, location and topic can reach the programme display while their hidden truth, producer notes and private performance instructions stay where they belong. We then bolted in tests to cover those boundaries.
We also added a mock voice provider, isolated verification scripts and private test routes, which meant Codex could exercise the important parts of the system without rinsing API credits, requesting a microphone or rearranging the development show I was actually using.
The local verification flow creates a temporary show, runs it from incoming to connected to interrupted to ended, checks the public snapshot, and deletes the temporary data afterwards – This is fun, so let make the ‘puter handle it!
These examples are what I now mean by harness design, it’s not one giant system prompt telling an agent to be excellent – It is the combination of context, constraints, interfaces, test data, state transitions, review points and safe ways to try things, all arranged so that a capable agent can do useful work without every task becoming an archaeological dig.
Taste is subtraction
The early caller tools were mega-comprehensive, which is a generous way of saying there were a ton of fields and things to click.
A caller can contain a public identity, a surface problem, a central desire, a worldview, actual behaviour, a contradiction, a hidden truth, escalation beats, suggested questions, voice choices, pace, style, response length, interruption behaviour, producer notes, topic tags, portraits and supporting visuals.
All of that information can be useful to the model and useful to the radio host producing, but putting it all in front of the person creating a caller made the experience feel weighty, and definetely not Apple-style joy.
So we simplified it. Then simplified it again….and then simplified it some more.
The AI caller builder now asks for one useful spark. You can optionally say what kind of call or tone you want, but you do not need to fill in a character dossier before anything interesting happens. Similarly, the workshop returns six different directions, you choose the one with life in it, and only then does it build the deeper production card behind that choice.
Manual creation went through the same treatment; The fields needed to get somebody on air come first, while the richer identity, behaviour, voice and graphics controls are available when they matter rather than standing in a row demanding attention. The caller library became searchable and filterable, with ready, draft and appeared states that answer practical production questions in an instant.
This was not about making the underlying system less capable. It was about preserving complexity underneath while making the working surface calmer – because a good production tool should make the operator feel more capable than the software looks.
Intent gates … coding bureaucracy
The AI workshop in the app can generate possibilities, but it cannot publish a caller. A generated caller is a draft. A producer can edit it, test it, add a portrait, prepare topic images and decide whether it is actually worth putting in front of a host. Only an approved caller can enter a live queue.
The same pattern works when building software with Codex; A plan is not a change, a change is not a verified result, and a passing test is not proof that something feels right when used. Then there is an intent gate, where I decide whether the proposed direction belongs in the product; an engineering gate, where tests and verification show whether the implementation holds together; and a product gate, where I use the thing and discover that the button I asked for is technically perfect but located somewhere near Belgium.
These gates make the work faster because they stop confident momentum in the wrong direction. Agents are extremely good at continuing. This is usually an advantage, although it can become a problem if the product has already wandered into a bar and got very drunk on it’s own idea and you have to tone it back down.
Trying the available voice models
Voice was where the project stopped being a normal web application and started developing opinions about microphones.
We tried four different routes: OpenAI Realtime, Gemini Live, Fish Audio and ElevenLabs Conversational AI. I am resisting the temptation to declare one universal winner because they have different strengths, different control surfaces and different ways of turning a perfectly ordinary pause into a small technical incident.
OpenAI Realtime became the default route in this build because it gave us a direct browser WebRTC flow, speech-to-speech interaction and useful semantic turn detection, while allowing the permanent API key to remain on the server. It felt like the most balanced starting point for the product, although tuning turn-taking still involved repeated conversations with fictional people in increasingly real arguments about fridges.
Gemini Live was useful as a comparison because it exposed more of the audio stream and voice activity configuration, which made it possible to experiment with speech-start sensitivity, speech-end sensitivity and silence duration – It also reminded us that provider behaviour is not exactly interchangeable. An instruction update that is harmless between turns can interrupt or disturb a response if it is applied at the wrong moment, so our Gemini adapter queues live-direction changes until the current answer has finished.
ElevenLabs is phenomenal with it’s enormous range of voices and strong control over voice character, speed and expressiveness, but it also works through a separately configured conversational agent, which means there is more account-side setup and another layer of behaviour to understand. That is not bad, it is simply a different thing to get your head around.
I gess the practical lesson was that the provider interface should not pretend these systems are identical – The application needs one stable language for the host, while the provider adapters contain the differences in authentication, audio formats, voices, interruptions, transcripts and session updates.
The host therefore chooses a route called OpenAI, Gemini, Fish or ElevenLabs, while the software worries about how that route actually works.
But, we are really, really, really waiting for full duplex
The current generation of API voice models is impressive, but most of the interaction still behaves as a sequence of turns: The host speaks, the system decides the host has finished, the caller replies, and the system decides what an interruption means, and so on.
That turn “boundary” is where an extraordinary amount of the awkwardness lives. And it is not good! A short thoughtful pause can look like the end of a sentence or background noise can look like an interruption – Two people beginning to speak at roughly the same time can cause voice-model uncertainty.
OpenAI’s new new new GPT-Live models are built on a full-duplex architecture, which means the model can listen and speak at the same time, continue listening while it responds, and make much more continuous decisions about whether to talk, wait, acknowledge or pause. GPT-Live is now powering ChatGPT Voice, and OpenAI says it plans to bring GPT-Live-1 and GPT-Live-1 mini to the API soon – We are waiting, yes please!!
For a phone-in, full duplex is not just a latency improvement, it changes the interaction model, so a caller could hear the host beginning to interject and naturally stop, acknowledge a brief “mmhm” without surrendering the entire turn, or perhaps deal and understand a presenter laughing, and understand that a pause is sometimes thinking rather than an invite to talkity talk.
The current system deliberately disables automatic noise-triggered barge-in and gives the host an explicit Interrupt control, because predictable control is better than theoretically natural chaos – When a proper full-duplex model is available through the API, I expect we will revisit that decision very enthusiastically.
OpenAI’s GPT-Live announcement describes the full-duplex interaction model and its planned API availability.
A show needed a workspace (not a settings page)
Another major shift was moving from one global studio into separate show workspaces. Yes! You can run multiple shows.
Each show now owns its title, format, caller guidance, voice route, running order, sound cues, output links and optional automation. A sports fan call-in show can have different caller behaviour from a late-night call-in show, and changing one show does not rewrite another one while nobody is looking.
The workspace separates prep from performance; The producer can arrange callers, reactivate somebody who has already appeared, add a newly approved caller during a live show, choose the broadcast layout and prepare sound cues, while the Host Studio remains focused on the immediate job of hosting.
That separation mattered because every control you add to a live screen competes with the caller, the audience and the presenter’s remaining brain capacity. The show workspace can contain the machinery, whilst the Host Studio should contain the next decision.
Simplify, simplify, simplify (again!)
The Host Studio went through multiple iterations because each version looked reasonable until I tried to use it as though I were actually presenting.
At first, controls accumulated wherever there was room, then we grouped them, then we reorganised them, then we removed steps and then we realised that a grouped collection of too many controls is still too many controls.
The current Studio centres the caller and the next action – If the show has not started, the main action is Start Show. If a caller is waiting, it is Answer, If the call needs a voice connection, it offers the chosen route, if the caller is live, the host gets interruption, mute, hold and end controls – and when a call finishes, the next caller is prepared automatically, but the host still decides exactly when they go on air.
Live tuning turned prompts into producer controls
Once the callers were working, another problem appeared: a caller that looked right on their caller card could still be erm, wrong in the moment!
Perhaps the call needed more energy, or perhaps the caller was becoming too intense and needed to de-escalate. Maybe they were answering coherently good but too long ? (which is a very realistic radio problem and also the working title of several meetings I have attended).
We added three live controls: Energy, Pace and Answer Length.
These do not permanently rewrite the caller, they are temporary producer nudges relative to the caller’s authored baseline, and they apply from the next response onwards. This enables the host to ramp the energy up, calm it down, make the delivery brisker, slow it down, request one concise sentence or allow a fuller answer, then reset everything for the next caller.
Underneath, the controls become provider-neutral instructions and the adapter applies them in a way that is safe for the active voice route. On the screen, however, they remain three sliders with labels such as Calmer, Livelier, Shorter and Fuller, because nobody hosting a show has ever thought, “What this moment needs is a textarea containing a revised system prompt.”
This is one of my favourite parts of the product because it demonstrates what the harness is for – The model remains generative and flexible, but the human gets a small, legible set of controls that match the job they are doing.
The broadcast output needed something to look at
Audio may be the heart of a phone-in, but we are also trying to keep viewers entertained, particularly the TikTok generation.
Each caller can therefore have a portrait and prepared supporting visuals. The producer can search stock providers such as Pexels or Pixabay, retain the photographer attribution, and trigger a relevant image manually during the call. A caller talking about cold-water swimming can show an icy lake. A caller arguing with a smart fridge can place an actual fridge on screen – This is haut couture.
We deliberately prepare visuals before the live call rather than firing off arbitrary image searches while the presenter is speaking (though I still may build this!).
The Host Studio reports the caller’s live output level to the broadcast page, so the display can include an audio equalizer without sending the host microphone through the public route. The output can run as an adaptive web pane, a 16:9 Twitch or OBS layout, a transparent overlay, or a portrait TikTok layout, because a design that looks splendid at 1920 by 1080 can look like a ransom note when squeezed into a phone.
Then I added an AI host, mostly because I could.
Once the system had AI callers, the obvious and slightly irresponsible question was: can an AI host could talk to them?
The answer is yes! The AI host can introduce a caller, respond after the caller finishes, close the conversation after a configured number of presenter turns, advance the queue and continue to the next caller, which means AI can now talk to AI forever, or at least until a provider limit, a database error or the heat death of the universe brings some editorial discipline to proceedings.
This is optional, but why not explore the boundaries?
So, what did I actually learn?
I started out wanting to make an AI radio phone-in and, along the way, ended up building the caller workshop, voice routes, running order, producer controls and broadcast output around it.
Codex made it possible to build and rebuild all of that ridiculously quickly. Similarly, the better the harness became, the more useful Codex became, but it still couldn’t tell me whether a caller felt alive, whether the host had too much to think about, or whether the latest impressive feature had actually made the product worse.
I only found those things by using it: sitting there with headphones on, talking to fictional callers, moving buttons and occasionally wondering why I had created a man who wanted to argue with me about his fridge.
That is probably the real shift, when software becomes much quicker to make, the difficult bit becomes deciding what deserves to exist, how it should behave and when you have added enough.
The studio is still an experiment, but it no longer feels like two AI voices joined to an API – It feels like somewhere I could sit down, put the headphones on and actually do the show.