I wanted to build a radio phone-in show where a human host could talk to fictional AI callers, which sounds quite straightforward until you remember that a radio phone-in is not simply two voices making noises at one another, because it also has a running order, a producer, callers waiting to come on, private notes, soundchecks, interruption, timing, jeopardy, graphics, audio levels, awkward silences, and the ever-present possibility that somebody’s microphone has quietly decided to become a decorative object.
Github: https://github.com/paolodit/phone-in-studio
The first version of the idea was not a specification. It was more like a pile of product thoughts that had been left in a carrier bag overnight. I wanted believable callers with different voices and different reasons for calling, I wanted to be able to test them privately, I wanted a host to bring them on air one at a time, I wanted a clean visual output for OBS and TikTok Live Studio, and I wanted the whole thing to feel like a piece of broadcasting software rather than an API demonstration wearing a lanyard.
Codex was very good at turning individual pieces of that mess into software, but the interesting part was not that it could produce React components, Prisma models or API routes quickly. The interesting part was learning how to build a harness around Codex so that its work kept moving towards the same product, and then building another harness inside the product so that unpredictable voice models could be used by somebody who was trying to host a show rather than conduct a live seminar on WebRTC.
That distinction became the whole project. The model was capable, the APIs were capable, and Codex was capable, but capability was not the scarce ingredient. The scarce ingredient was taste, by which I mean deciding what the software should feel like, what the host should be thinking about at any particular moment, what should happen automatically, what should require a deliberate click, and what should be removed even though it had taken an annoyingly impressive amount of work to build.
I did not start with a perfect brief
I use Codex most effectively when I start by telling it what I am actually trying to achieve, including the scrappy bits, the contradictions and the things I have not decided yet, rather than pretending I have arrived with a fully formed technical specification delivered on a stone tablet.
For this project, the important sentence was not “build an AI voice application.” It was “I want to host a radio phone-in.” That immediately changes the shape of the work, because a host does not want to manage session tokens, inspect JSON, choose a voice activity detection threshold or wonder whether an audio buffer has been cleared. The host wants to see who is calling, understand why they are calling, answer them, interrupt them, put them on hold, bring in the next person and keep the programme moving.
I would give Codex a broad product goal, ask it to inspect the repository, turn the idea into a plan, identify the decisions that would be expensive to reverse, and then challenge parts of the plan that felt technically respectable but experientially wrong. Sometimes this was a carefully written prompt and sometimes it was me dictating a stream of thoughts that began with a sensible observation about the running order and ended with a complaint about why a particular button was three inches too far away. Both approaches worked, although only one makes me sound organised in an article.
The important thing was that the conversation produced durable structure. Caller, candidate, approved caller, queue item, incoming, connecting, live, held and ended became real product concepts. Private production notes and public broadcast information became separate data boundaries. Voice providers became replaceable implementations behind one interface. The show lifecycle became a state machine rather than a collection of buttons that all hoped the other buttons had behaved themselves.
This was the first part of the harness: giving Codex a stable vocabulary, a repository structure and a definition of what done meant, so that each change did not have to rediscover the entire product from first principles.
The harness gradually became executable
At the beginning, much of the guidance lived in conversations and an implementation plan, but the best rules gradually stopped being prose and became code that could enforce the decision.
The show state machine knows that a caller cannot be answered before they are incoming, that a caller cannot be put on hold before they are live, and that the next caller cannot be cued when there is nobody left in the queue. The broadcast serializer uses an allow-list, so a caller’s public name, location and topic can reach the programme display while their hidden truth, producer notes and private performance instructions stay where they belong. Tests cover those boundaries, because “please remember not to reveal the secret bit live on air” is not a security strategy, even if you say please twice.
We also added a mock voice provider, isolated verification scripts and private test routes, which meant Codex could exercise the important parts of the system without spending API credit, requesting a microphone or rearranging the development show I was actually using. The local verification flow creates a temporary show, runs it from incoming to connected to interrupted to ended, checks the public snapshot, and deletes the temporary data afterwards. This is less glamorous than an AI caller arguing with a presenter, but it is also less likely to leave the database looking as though a small radio station has been burgled.
That is what I now mean by harness design. It is not one giant system prompt telling an agent to be excellent. It is the combination of context, constraints, interfaces, test data, state transitions, review points and safe ways to try things, all arranged so that a capable agent can do useful work without every task becoming an archaeological dig.
Taste mostly appeared as subtraction
The early caller tools were comprehensive, which is a generous way of saying there were lots of fields.
A caller can contain a public identity, a surface problem, a central desire, a worldview, actual behaviour, a contradiction, a hidden truth, escalation beats, suggested questions, voice choices, pace, style, response length, interruption behaviour, producer notes, topic tags, portraits and supporting visuals. All of that information can be useful to the model and useful to the production, but putting it all in front of the person creating a caller made the experience feel like completing an insurance form for somebody who did not technically exist.
So we simplified it, then simplified it again, and then simplified it some more, because apparently the first two simplifications were only warming up.
The AI caller builder now asks for one useful spark. You can optionally say what kind of call or tone you want, but you do not need to fill in a character dossier before anything interesting happens. The workshop returns six different directions, you choose the one with life in it, and only then does it build the deeper production card behind that choice.
Manual creation went through the same treatment. The fields needed to get somebody on air come first, while the richer identity, behaviour, voice and graphics controls are available when they matter rather than standing in a row demanding attention. The caller library became searchable and filterable, with ready, draft and appeared states that answer practical production questions quickly.
This was not about making the underlying system less capable. It was about preserving complexity underneath while making the working surface calmer, because a good production tool should make the operator feel more capable than the software looks.
Approval gates were not bureaucracy
The caller workflow ended up mirroring the Codex workflow.
The AI workshop can generate possibilities, but it cannot publish a caller. A generated caller is a draft. A producer can edit it, test it, add a portrait, prepare topic images and decide whether it is actually worth putting in front of a host. Only an approved caller can enter a live queue.
The same pattern works when building software with Codex. A plan is not a change, a change is not a verified result, and a passing test is not proof that something feels right when used. There is an intent gate, where I decide whether the proposed direction belongs in the product; an engineering gate, where tests and verification show whether the implementation holds together; and a product gate, where I use the thing and discover that the button I asked for is technically perfect but located somewhere near Belgium.
These gates make the work faster because they stop confident momentum in the wrong direction. Agents are extremely good at continuing. This is usually an advantage, although it can become a problem if the product has already wandered into a hedge.
Trying the available voice models
Voice was where the project stopped being a normal web application and started developing opinions about microphones.
We tried three different routes: OpenAI Realtime, Gemini Live and ElevenLabs Conversational AI. I am resisting the temptation to declare one universal winner because they have different strengths, different control surfaces and different ways of turning a perfectly ordinary pause into a small technical incident.
OpenAI Realtime became the default route in this build because it gave us a direct browser WebRTC flow, speech-to-speech interaction and useful semantic turn detection, while allowing the permanent API key to remain on the server. It felt like the most balanced starting point for the product, although tuning turn-taking still involved repeated conversations with fictional people in increasingly real arguments about fridges.
Gemini Live was useful as a comparison because it exposed more of the audio stream and voice activity configuration, which made it possible to experiment with speech-start sensitivity, speech-end sensitivity and silence duration. It also reminded us that provider behaviour is not interchangeable. An instruction update that is harmless between turns can interrupt or disturb a response if it is applied at the wrong moment, so our Gemini adapter queues live-direction changes until the current answer has finished.
ElevenLabs brought a different proposition, with an enormous range of voices and strong control over voice character, speed and expressiveness, but it also works through a separately configured conversational agent, which means there is more account-side setup and another layer of behaviour to understand. That is not bad, it is simply a different bargain.
The practical lesson was that the provider interface should not pretend these systems are identical. The application needs one stable language for the host, while the provider adapters contain the differences in authentication, audio formats, voices, interruptions, transcripts and session updates.
The host therefore chooses a route called OpenAI, Gemini or ElevenLabs, while the software worries about how that route actually works. I consider this a fair division of labour.
We are really, really waiting for full duplex
The current generation of API voice models is impressive, but most of the interaction still behaves as a sequence of turns. The host speaks, the system decides the host has finished, the caller replies, and the system decides what an interruption means.
That turn boundary is where an extraordinary amount of the awkwardness lives. A short thoughtful pause can look like the end of a sentence. Background noise can look like an interruption. Two people beginning to speak at roughly the same time can produce the conversational grace of two supermarket trolleys meeting in a narrow aisle.
OpenAI’s new GPT-Live models are built on a full-duplex architecture, which means the model can listen and speak at the same time, continue listening while it responds, and make much more continuous decisions about whether to talk, wait, acknowledge or pause. GPT-Live is now powering ChatGPT Voice, and OpenAI says it plans to bring GPT-Live-1 and GPT-Live-1 mini to the API soon. We are waiting for that API release with what I would describe as professional interest, if professional interest normally involves refreshing an announcement page while whispering “come on” at it.
For a phone-in, full duplex is not just a latency improvement. It changes the interaction model. A caller could hear the host beginning to interject and naturally stop, acknowledge a brief “mmhm” without surrendering the entire turn, cope with a presenter laughing, and understand that a pause is sometimes thinking rather than an invitation to deliver another paragraph.
The current system deliberately disables automatic noise-triggered barge-in and gives the host an explicit Interrupt control, because predictable control is better than theoretically natural chaos. When a proper full-duplex model is available through the API, I expect we will revisit that decision very enthusiastically, probably on the same day.
OpenAI’s GPT-Live announcement describes the full-duplex interaction model and its planned API availability.
A show needed a workspace, not a settings page
Another major shift was moving from one global studio into separate show workspaces.
Each show now owns its title, format, caller guidance, voice route, running order, sound cues, output links and optional automation. A sports fan line can have different caller behaviour from a late-night story programme, and changing one show does not quietly rewrite another one while nobody is looking.
The workspace separates preparation from performance. The producer can arrange callers, reactivate somebody who has already appeared, add a newly approved caller during a live show, choose the broadcast layout and prepare sound cues, while the Host Studio remains focused on the immediate job of hosting.
That separation mattered because every control you add to a live screen competes with the caller, the audience and the presenter’s remaining brain capacity. The show workspace can contain the machinery. The Host Studio should contain the next decision.
Simplify, simplify, simplify
The Host Studio went through multiple iterations because each version looked reasonable until I tried to use it as though I were actually presenting.
At first, controls accumulated wherever there was room. Then we grouped them. Then we reorganised them. Then we removed steps. Then we discovered that a grouped collection of too many controls is still too many controls, only now they have matching borders.
The current Studio centres the caller and the next action. If the show has not started, the main action is Start Show. If a caller is waiting, it is Answer. If the call needs a voice connection, it offers the chosen route. If the caller is live, the host gets interruption, mute, hold and end controls. When a call finishes, the next caller is prepared automatically, but the host still decides exactly when they go on air.
That last detail is a small example of taste doing more work than technology. Fully manual queue progression added a redundant step at the busiest moment. Fully automatic progression would take editorial timing away from the host. Preparing the next caller automatically while keeping the actual answer under human control gives us the useful half of automation without the haunted-radio-station half.
Live tuning turned prompts into producer controls
Once the callers were working, another problem appeared: a caller that looked right on their saved card could still be slightly wrong in the moment.
Perhaps the call needed more energy. Perhaps the caller was becoming too intense and needed to de-escalate before everybody started writing to Ofcom. Perhaps the answers were good but too long, which is a very realistic radio problem and also the working title of several meetings I have attended.
We added three live controls: Energy, Pace and Answer Length.
These do not permanently rewrite the caller. They are temporary producer nudges relative to the caller’s authored baseline, and they apply from the next response onwards. The host can ramp the energy up, calm it down, make the delivery brisker, slow it down, request one concise sentence or allow a fuller answer, then reset everything for the next caller.
Underneath, the controls become provider-neutral instructions and the adapter applies them in a way that is safe for the active voice route. On the screen, however, they remain three sliders with labels such as Calmer, Livelier, Shorter and Fuller, because nobody hosting a show has ever thought, “What this moment needs is a textarea containing a revised system prompt.”
This is one of my favourite parts of the product because it demonstrates what the harness is for. The model remains generative and flexible, but the human gets a small, legible set of controls that match the job they are doing.
The broadcast output needed something to look at
Audio may be the heart of a phone-in, but we are also trying to keep viewers entertained, particularly the TikTok generation, whose attention is apparently leased by the second and may be repossessed without notice.
Each caller can therefore have a portrait and prepared supporting visuals. The producer can search stock providers such as Pexels or Pixabay, retain the photographer attribution, and trigger a relevant image manually during the call. A caller talking about cold-water swimming can show an icy lake. A caller arguing with a smart fridge can place an actual fridge on screen. This is high culture.
We deliberately prepare visuals before the live call rather than firing off arbitrary image searches while the presenter is speaking. The host can choose when an image appears, remove it, or remain on the caller portrait, and the broadcast output receives only the public caller information and the selected asset.
The Host Studio reports the caller’s live output level to the broadcast page, so the display can include an audio equalizer without sending the host microphone through the public route. The output can run as an adaptive web pane, a 16:9 Twitch or OBS layout, a transparent overlay, or a portrait TikTok layout, because a design that looks splendid at 1920 by 1080 can look like a ransom note when squeezed into a phone.
Then I added an AI host, mostly because it amused me
Once the system had AI callers, the obvious and slightly irresponsible question was whether an AI host could talk to them.
The answer is yes. The AI host can introduce a caller, respond after the caller finishes, close the conversation after a configured number of presenter turns, advance the queue and continue to the next caller, which means AI can now talk to AI forever, or at least until a provider limit, a database error or the heat death of the universe brings some editorial discipline to proceedings.
I did not add this because I think fully automated radio is the main use of the project. I added it because it was funny, technically interesting, and a useful test of whether the harness could support more autonomy without surrendering control.
The AI Host and Caller Factory are therefore optional modules and hidden in a fresh installation. They must be enabled globally, then enabled again for an individual show. Auto-run never arms itself when the page loads. The host can take over, pause it or hit Emergency Stop, and a generation, speech or transition error pauses the automation rather than allowing it to improvise its way through a small fire.
This is another place where the product and the development process rhyme. Greater capability needs stronger boundaries, not fewer. The more the system can do by itself, the clearer the stop conditions, approval gates and recovery controls need to become.
The trials were where the product emerged
The repository history looks like a series of features, but the actual process felt more like a repeated collision between an idea and the physical world.
Microphone security rules meant that something working on localhost could fail on a plain HTTP LAN address. Audio routing meant we had to be explicit about what the browser played, what OBS captured and why the host should wear headphones. Automatic barge-in sounded natural in theory but became too sensitive to room noise. Voice providers interpreted pauses, interruptions and updated instructions differently. Rich caller tools became cluttered. A useful running order needed replay and reactivation controls. A broadcast display needed to reconnect to the current state rather than replaying old events. Tests needed isolated data so that verifying the app did not alter the show we wanted to demonstrate.
None of those lessons came from asking Codex to write more code. They came from using the software, noticing where my attention went, describing the problem precisely, and then giving Codex a better target and a better way to verify it.
This is why I do not think the most useful description is “AI built this app.” Codex wrote and rewrote a substantial amount of the implementation, traced changes across the repository, converted constraints into tests, maintained the provider boundaries and helped turn each awkward observation into a concrete change, but the direction came from repeatedly deciding what kind of product this was.
Codex could tell me how to add another control. Taste told me that the correct answer was sometimes to remove two.
What I would carry into the next project
The biggest lesson is that an agent becomes more useful as the workspace around it improves.
Give it the real goal rather than only the immediate ticket. Give the repository stable product language. Turn important decisions into types, state machines and allow-lists. Create mocks for expensive or physical dependencies. Make verification isolated and repeatable. Separate deterministic checks from real-world rehearsals. Review the work at the level of product behaviour, not just whether the code compiles. When the same mistake or correction appears twice, promote the lesson out of the conversation and into durable repository guidance, a test or a script.
Most importantly, keep using the thing.
A technically correct voice session is not necessarily a good call. A comprehensive production card is not necessarily a usable caller builder. A powerful automated host is not necessarily something that should greet every user on the home screen. A beautiful wide broadcast layout is not necessarily readable on TikTok. A button can be valid, accessible and still be in completely the wrong place when somebody is trying to end a live call.
Codex changed the economics of implementing and revising the software, which meant I could explore more ideas and correct more mistakes without each iteration becoming a small funding round, but faster implementation only made taste more important because there were now many more plausible directions available.
The harness made Codex effective. Taste determined what effectiveness meant.
And the result, after a great deal of building, testing, simplifying, listening, interrupting and moving buttons around, is beginning to feel less like an AI voice demo and more like somewhere you could actually sit down, put your headphones on, open the line and host a show.
Leave a Reply