Interact-0 is a full-duplex audio model with 31B parameters, 3B active. We trained it on 9 million hours of single-channel audio, and 500,000 hours of channel-separated multi-speaker audio. The model is composed of an audio encoder, a thinking trunk, and text and audio output heads.
Architecture, Single-Channel Audio Continuation
The encoder turns audio into features for the trunk. The output heads predict text tokens and audio codebooks; only one of the two output heads is used at each position.
Audio is split into 80ms ticks. We first trained the model for single-channel pure audio continuation. Then we introduced text planning, allowing the model to reason about what it will say ahead of time. Finally, we trained the model to predict two simultaneous audio streams: one for the user and one for the model.
Text Planning and Full-Duplex Token Orders
Aa
Tick
Aa
Tick
Aa
Tick
Aa
Tick
AaAa
Tick
AaAa
Tick
AaAa
Tick
Above: Alternating text and audio positions allow the model to reason about what it will say before it says it. Below: Each tick contains four positions, in order: User audio → User text → Model text → Model audio.
Pre-Training
We initialized from the weights of Qwen3-Omni [1]. It was designed to take in audio clips all at once, so the attention in its audio encoder is bidirectional. We require a streaming audio input, so we first trained its audio encoder to instead use causal attention. We froze the main trunk and trained only the encoder on ASR tasks until the word error rate surpassed that of Qwen3-Omni itself.
Once we had a causal encoder, we trained the thinking trunk. We trained on 9 million hours of diverse speech data, with strict filters for audio and speech quality. In this stage, we kept an anchor loss on text to prevent the model from losing its verbal intelligence, using a mixed objective of 90% KL divergence against Qwen3-Base and a 10% CE loss on a diverse corpus of text pretraining data.
Speech continuation loss
Held-out validation loss from the pure audio continuation run.
1.6 s · Outputs begin
Prompt
0 s5 s10 s15 s21.6 s
We created a set of internal benchmarks testing consistency, world knowledge, and logical reasoning.
Internal benchmark accuracy
Share of answers marked correct over the course of the pure audio continuation run.
During pure audio training, improvements were largely in audio quality, voice consistency, and word and sentence formation rather than intelligence.
Speaker similarity across checkpoints
Each dot is one continuation of the strawberry prompt shown above, at different stages of the pure audio continuation run. We measured ECAPA cosine similarity between different four-second speech windows within the continuation and the prompt voice.
Adding Text Planning
To improve our model’s intelligence, we added text planning. This allows the model to reason about what it will say before speaking. Positions alternate between text tokens and an audio token, with each text token appearing slightly before it is spoken. Because text tokens on average take more than one 80ms audio tick to speak, many of the text token slots are empty. In these slots we use PAD and EPAD tokens in the style of Moshi [2].
Text alignment
Text tokens in training rows were aligned to audio, given a 100ms buffer before the corresponding audio, and quantized to the 80ms grid.
To train text planning in our model, we took 2.5 million hours of our higher-quality data, transcribed it, and aligned the words to audio. On audio positions, we maintained pure cross-entropy loss. On text positions, we used a mixture of KL divergence against Qwen3-Base given the text stream, and cross-entropy on the true text token.
We started out with a mix of 90% KL and 10% CE. But since spoken and written language follow different distributions, our model quickly surpassed the teacher at predicting the next text token during speech, so we lowered and then removed the KL loss component entirely.
Text planning loss
Held-out text token cross-entropy during the text planning run. Left: steps 0–4,000. Right: steps 4,000 onward on a tighter loss scale.
1.6 s · Outputs begin
Prompt
0 s5 s10 s15 s21.6 s
Using the same internal benchmark as before, we can see how text planning dramatically increases the intelligence of the model (text planning checkpoints in red).
Internal benchmark accuracy
Share of answers marked correct over the course of the pure audio continuation and text planning runs.
Towards Full-Duplex
Duplex pre-training requires large quantities of multi-speaker data, with a separate audio channel for each speaker. Very little of this data is publicly available online. Segmentation models have been trained to split single-channel audio into separate channels for each speaker, but none of them, neither open nor closed, met our bar.
So we took a detour to train our own in-house speech segmentation model:
Speaker separation performance
Model
Mean SI-SNR (dB)Higher is better
Overall WERLower is better
Ours
18.6
4.0%
AudioShake
11.8
10.1%
MossFormer2
10.5
20.6%
TISDiSS
2.1
54.0%
SoloSpeech
1.7
46.8%
Performance on held-out data. WER refers to the word error rate of the transcribed audio of the separated channels.
Our segmentation model is also able to split audio with more than two speakers into several separated channels, something that none of the other models can do effectively, which gives us the data necessary to train our model to act as one person among many in group conversations.
For duplex training rows, we use four positions at each tick: user audio, user text, model text, and model audio. We initialized four additive embedding vectors close to zero, one for each type. To adjust the model to this paradigm, we trained on 500k hours of single-speaker audio on only one of either the user or the model side, while keeping the other side silent.
Four Positions per Tick
User audioAaUser textAaModel textModel audio
Tick
User audioAaUser textAaModel textModel audio
Tick
When training the text stream on the user side, we placed text tokens after their corresponding audio has been said, in order to establish a transcription-style prior for the user text stream. We trained on single-speaker audio until the model-side loss reached that of our text planning model, and until the user-side word error rate was close to that of frontier transcription models.
Duplex text alignment
In duplex training rows, model text tokens were given a 100ms buffer before the corresponding audio and user text tokens a 100ms buffer after it. Both were then quantized to the 80ms grid.
To compensate for the real-world delay during inference for on-device audio processing, computation, and network, we introduced a delay of three ticks (240ms) in the user rows seen by the model. We then trained on 500,000 hours of segmented audio using this setup.
Full-duplex loss
Held-out validation loss on duplex data. The sharp drops in model text loss occurred when we dropped the reliance on a text-model-based teacher, allowing our model to pick up more speech-specific patterns.
The clip below features two instances of our pre-trained duplex model speaking with each other. Speaker 1 opens with a recorded greeting and Speaker 2 opens with the strawberry prompt.
4.72 s · Prefix ends
Speaker 1 (left)Speaker 2 (right)
0 s10 s20 s30 s44.54 s
Our base model behaves more like the average person than an assistant. If you start talking about movies, it might start talking about food. With no reason to follow what the user says, it often refuses questions, claims to not know answers, or goes off on tangents. We post-trained the model to adjust this behavior.
Post-Training
In existing speech models, post-training is a tradeoff between desired behavior and naturalness. If you post-train too heavily, you end up with a model that has lost the human-like feel that it started with. If you don’t post-train enough, the model is unusable in any practical setting.
2.08 s · Prefix ends
0 s10 s20 s29.28 s
We sought to strike a balance here, applying a light touch to our post-training and targeting specific, carefully-chosen behaviors using Reinforcement Learning. Our final RL pipeline consisted of only around 1,300 hours of rollouts.
RL reward
Reward at each step over the course of our final RL pipeline.
In the early stages of RL, we played fixed prefixes against our model and rewarded on-topic and timely responses using GRPO. Once we saturated the reward against these fixed prefixes, we trained on a form of modified self-play, where our post-trained model speaks to another instance of the model.
This required us to build a good “user simulator” for our model to talk to. In text-based language models, user simulators are something that people have been researching for a long time. But there is nothing of the sort for full-duplex models. So we post-trained a version of our base model to adopt the user side of user-model interactions, giving our model a realistic conversation partner for RL rollouts. This allows us to better post-train on multi-speaker scenarios.
Multi-speaker RL reward
Reward at each epoch across six consecutive RL runs on multi-speaker conversations, with held-out reward shown as dots. Dashed lines divide the runs, where scenario difficulty increases with each run, causing the sharp drops in reward.
We also trained checkpoints to be able to take in a text prefix, what we call a “reference document,” as additional input before a conversation. This allows us to provide the model with external information, hook it up to task-specific harnesses we’ve built, and compact conversations so that they can last longer than our native context window.
We encountered some surprising generalization results during reinforcement learning. During one training run, we performed self-play rollouts with our latest blog post as the reference document, on prefixes with the user debating the model about the blog. Despite only training on this one essay, this training generalized such that the model could debate competently about any article of similar length, with no leakage of information from our blog post:
Tanner (left)Will (right)
0 s10 s20 s30 s40 s54.91 s
Results
Interact-0 is the only model that can behave naturally in group settings. Our checkpoint, named Orchestra, is trained to stay quiet most of the time, listening in on the conversation and chiming in when it has something useful to provide, or when it is called on.
Because no existing models can interact naturally in groups, there are no benchmarks for this. We created our own benchmarks to fill this need: Multi-Bench and Interruption-Bench.
Multi-Bench sets up scenarios where the model is in a conversation with two other people. The model must understand when the other people are talking to it, versus when they’re talking to each other, and respond or remain silent accordingly. In some scenarios, the model simply has to respond to its name; in others, it must decide based on context clues.
Interruption-Bench tracks the ability of a model to keep talking when the user says backchannels such as “uh-huh,” “yeah,” “go on,” during its response. When models are comfortable holding the floor, they allow users to freely express themselves during the conversation without worrying that the model will cut itself off.
Conversation benchmark results
Model
Multi-BenchHigher is better
Interruption-BenchHigher is better
Interact-0
74.2%
65%
GPT Live 1
1.5%
32%
Gemini 3.8 Live
4.5%
1%
Phonic 1
3.5%
52%
Multi-Bench reports the share of episodes passed. Interruption-Bench reports the share of user backchannels the model kept talking through. Interact-0 is our Orchestra checkpoint on Multi-Bench and our Duet checkpoint on Interruption-Bench.
Model: LeilaSpeakStay quiet
Expected
JessMarco
0 s20 s40 s67 s
Model: EmmettSpeakStay quiet
Expected
HollyRaj
0 s20 s40 s67 s
User
0 s5 s10 s15.7 s
User
0 s5 s10 s15.5 s
Hobblings
We are hobbled by hockey and power outages. We trained Interact-0 anyway.
Our data guy sustained a concussion from a hockey game right before some of our most important post-training runs.
Our office suffered a sudden power outage last week, so we had to move our local Inference Server to our investors' office for the weekend.