Digital audio workstations still run on a visual metaphor borrowed from magnetic tape: horizontal strips, a single playhead, everything locked to a track. For thirty years, this setup has guided the way music is recorded. But a growing body of work on real-time generative audio and latency-sensitive models are changing that. Instead of just playing back sounds on command, they can listen, respond, and play along with musicians in real time, acting more like a bandmate than a tool.
Paul Smith, a Product Designer at Google Cloud who focuses on AI agents, is also a trained orchestral conductor. His career includes building adaptive AI for Amazon Q Business and leadership roles at Pearson and Tanium. About ten years ago, working out of the Harvard Innovation Lab, he helped build Cadenza, an app that used early predictive models to follow a live musician and adjust orchestral accompaniment in real time. That experience inspired his belief that systems designed for interaction, not just output, will drive the next wave of creative technology.
"The real opportunity is to make live music more responsive, fluid, and musical. The future isn’t about creating better tools as much as about removing the tool layer altogether," explains Smith. He believes that many efforts focus on producing convincing tracks, but the bigger transformation happens during the act of making music, not just the output.
- Stuck in the grid: Smith believes the DAW interface has barely changed in three decades. He sees it as a lens that can distort how people approach creation. "The screen shows rectangles across strips, like music tracks on audio tape," he explains. "It’s an artificial lens for thinking about music-making." Traditional recording workflows amplify this effect by isolating players in separate rooms and routing performance through headphones, prioritizing software logic over the spontaneity of playing together.
- Less work, more play: Smith’s design philosophy starts with an observation about human behavior that he has seen in every field he has worked in. "People don’t want extra work. They want technology to take work away," he says. Much of today’s music software does the opposite, rewarding those who master its complexity while leaving others behind. Knowing the tool becomes a mark of skill, but the tool itself can stand between the creator and the creative act.
Smith’s work on Cadenza shows how music AI can respond naturally to a performer. The app listened to a student playing a flute part, detected whether they were slowing down or speeding up, and adjusted a full orchestral accompaniment by John Williams to match, without any configuration or button presses. Over time, the team removed every unnecessary step until the app could hear the first note, identify the piece, and start playing alongside the musician automatically.
- Fade to focus: That kind of responsive interaction is Smith’s design benchmark. When Cadenza worked, musicians stopped noticing it entirely. "People enter the zone and make music without thinking about the technology," he says. "It fades into the background because it only adds musical context." He compares it to driving: "When we brake, the automatic braking system in the car engages automatically to avoid an obstacle. That control feels natural. That’s the kind of partnership we want in music."
- Bandmate advisor: Beyond synchronization, Smith sees AI as a real-time musical advisor, listening to a performer and suggesting alternatives like a bandmate in rehearsal. "AI can offer a new perspective on how a phrase is played," he says. "It responds in the moment and helps improve artistic choices without taking them over."
Smith envisions AI handling the technical tasks of live performance, like adjusting levels, separating parts, and responding to cues in real time. When a musician’s hands are tied, the intelligent system can step in for tasks that would normally need extra people or pauses in the performance.
- Ghost in the mixer: Live performance already relies on a sound engineer for mixing. Smith sees AI absorbing that role in a smooth, context-aware way. "AI can listen, notice when a solo is taking the lead, and balance the other parts automatically," he says. "It mixes spontaneous music in real time." New advances in source separation and live mixing are already moving toward this kind of capability.
- Milliseconds over minutes: Creating a responsive system requires latency low enough for AI to sound like it’s playing at the same time. Smith sees the closing gap between generation speed and real-time playback as a major breakthrough. "AI can now respond instantly to how music is played," he says. "Generating in milliseconds instead of seconds makes it feel like a real-time interaction."
Smith sees DAWs as tools for a specialized professional audience. He is more interested in the impact of AI for the millions who make music without ever opening production software. Experiments at SXSW and in everyday audio apps suggest these systems could fit naturally into creative life without needing new interfaces.
- Choir power: For choir singers, students, and hobbyists, the challenge has always been the interface, not talent. "Think about the 50 million people singing in US choirs compared with DAW users," he says. "Technology that listens and responds automatically can reach far more people and have a bigger impact than DAWs ever will."
- The encore effect: Smith applies this idea to any medium where humans perform spontaneously and systems respond. Once AI can create media in real time alongside human input, the same approach works for immersive environments, games, and film. "A human performs, and the system detects, interprets, and enhances the output in real time," he explains.
Smith emphasizes building technology for the moment of creation, not just the final product. The most successful tools empower creativity without getting in the way, letting musicians focus on expression rather than mechanics. He envisions a future where AI disappears into the background, responding naturally, amplifying ideas, and enabling musicians and creators to do more than they could alone. The real breakthrough comes when technology stops feeling like a tool and starts feeling like a partner. "When designed for interaction, not output, few realize the scale of what this can become once it starts working," he concludes.