interact

Interact-0

Architecture

Interact-0 is a full-duplex audio model with 31B parameters, 3B active. We trained it on 9 million hours of single-channel audio, and 500,000 hours of channel-separated multi-speaker audio. The model is composed of an audio encoder, a thinking trunk, and text and audio output heads.

Architecture, Single-Channel Audio Continuation
Text is encoded and fed back into the trunk.Audio frame80 msAudio encoderCausal attentionThinking trunk30B MoE TransformerText headAudio headText tokens16 RVQ codebooksGenerated speech is fed back as the model’s own audio history.
The encoder turns audio into features for the trunk. The output heads predict text tokens and audio codebooks; only one of the two output heads is used at each position.

Audio is split into 80ms ticks. We first trained the model for single-channel pure audio continuation. Then we introduced text planning, allowing the model to reason about what it will say ahead of time. Finally, we trained the model to predict two simultaneous audio streams: one for the user and one for the model.

Text Planning and Full-Duplex Token Orders
Above: Alternating text and audio positions allow the model to reason about what it will say before it says it. Below: Each tick contains four positions, in order: User audio → User text → Model text → Model audio.

Pre-Training

We initialized from the weights of Qwen3-Omni [1]. It was designed to take in audio clips all at once, so the attention in its audio encoder is bidirectional. We require a streaming audio input, so we first trained its audio encoder to instead use causal attention. We froze the main trunk and trained only the encoder on ASR tasks until the word error rate surpassed that of Qwen3-Omni itself.

Once we had a causal encoder, we trained the thinking trunk. We trained on 9 million hours of diverse speech data, with strict filters for audio and speech quality. In this stage, we kept an anchor loss on text to prevent the model from losing its verbal intelligence, using a mixed objective of 90% KL divergence against Qwen3-Base and a 10% CE loss on a diverse corpus of text pretraining data.

Speech continuation loss
Speech continuation validation loss across two consecutive training runs, starting at step 2,000.
Held-out validation loss from the pure audio continuation run.
1.6 s · Outputs begin
Prompt

0:00 / 21.6 s

We created a set of internal benchmarks testing consistency, world knowledge, and logical reasoning.

Internal benchmark accuracy
Internal benchmark accuracy by checkpoint upload date.
Share of answers marked correct over the course of the pure audio continuation run.

During pure audio training, improvements were largely in audio quality, voice consistency, and word and sentence formation rather than intelligence.

Speaker similarity across checkpoints
Speaker similarity across three audio-only checkpoints. Median scores rise from 0.385 to 0.539 to 0.591, with 48 continuations per checkpoint.
Each dot is one continuation of the strawberry prompt shown above, at different stages of the pure audio continuation run. We measured ECAPA cosine similarity between different four-second speech windows within the continuation and the prompt voice.

Adding Text Planning

To improve our model’s intelligence, we added text planning. This allows the model to reason about what it will say before speaking. Positions alternate between text tokens and an audio token, with each text token appearing slightly before it is spoken. Because text tokens on average take more than one 80ms audio tick to speak, many of the text token slots are empty. In these slots we use PAD and EPAD tokens in the style of Moshi [2].

Text alignment
Before the wordSpeechModel plan100 ms buffer“hello”word audio
Text tokens in training rows were aligned to audio, given a 100ms buffer before the corresponding audio, and quantized to the 80ms grid.

To train text planning in our model, we took 2.5 million hours of our higher-quality data, transcribed it, and aligned the words to audio. On audio positions, we maintained pure cross-entropy loss. On text positions, we used a mixture of KL divergence against Qwen3-Base given the text stream, and cross-entropy on the true text token.

We started out with a mix of 90% KL and 10% CE. But since spoken and written language follow different distributions, our model quickly surpassed the teacher at predicting the next text token during speech, so we lowered and then removed the KL loss component entirely.

Text planning loss
Text planning validation word-token loss. The left panel shows steps 0–4,000, followed by a scale break and a larger zoomed panel from step 4,000 onward. No interval is omitted.
Held-out text token cross-entropy during the text planning run. Left: steps 0–4,000. Right: steps 4,000 onward on a tighter loss scale.
1.6 s · Outputs begin
Prompt

0:00 / 21.6 s

Using the same internal benchmark as before, we can see how text planning dramatically increases the intelligence of the model (text planning checkpoints in red).

Internal benchmark accuracy
Internal benchmark accuracy by checkpoint upload date. Checkpoints with text planning are highlighted in red.
Share of answers marked correct over the course of the pure audio continuation and text planning runs.

Towards Full-Duplex

Duplex pre-training requires large quantities of multi-speaker data, with a separate audio channel for each speaker. Very little of this data is publicly available online. Segmentation models have been trained to split single-channel audio into separate channels for each speaker, but none of them, neither open nor closed, met our bar.

So we took a detour to train our own in-house speech segmentation model:

Speaker separation performance
ModelMean SI-SNR (dB)Higher is betterOverall WERLower is better
Ours18.64.0%
AudioShake11.810.1%
MossFormer210.520.6%
TISDiSS2.154.0%
SoloSpeech1.746.8%
Performance on held-out data. WER refers to the word error rate of the transcribed audio of the separated channels.

Our segmentation model is also able to split audio with more than two speakers into several separated channels, something that none of the other models can do effectively, which gives us the data necessary to train our model to act as one person among many in group conversations.

For duplex training rows, we use four positions at each tick: user audio, user text, model text, and model audio. We initialized four additive embedding vectors close to zero, one for each type. To adjust the model to this paradigm, we trained on 500k hours of single-speaker audio on only one of either the user or the model side, while keeping the other side silent.

Four Positions per Tick

When training the text stream on the user side, we placed text tokens after their corresponding audio has been said, in order to establish a transcription-style prior for the user text stream. We trained on single-speaker audio until the model-side loss reached that of our text planning model, and until the user-side word error rate was close to that of frontier transcription models.

Duplex text alignment
Before the wordSpeechModel plan100 ms buffer“hello”word audioAfter the wordUser transcriptword audio“hello”100 ms buffer
In duplex training rows, model text tokens were given a 100ms buffer before the corresponding audio and user text tokens a 100ms buffer after it. Both were then quantized to the 80ms grid.

To compensate for the real-world delay during inference for on-device audio processing, computation, and network, we introduced a delay of three ticks (240ms) in the user rows seen by the model. We then trained on 500,000 hours of segmented audio using this setup.

Full-duplex loss
Four duplex validation curves in a two-by-two grid. Model audio and model text are on top, with user text and duplex total below. The dashed line at step 4,000 marks the omitted initial interval.
Held-out validation loss on duplex data. The sharp drops in model text loss occurred when we dropped the reliance on a text-model-based teacher, allowing our model to pick up more speech-specific patterns.

The clip below features two instances of our pre-trained duplex model speaking with each other. Speaker 1 opens with a recorded greeting and Speaker 2 opens with the strawberry prompt.

4.72 s · Prefix ends
Speaker 1 (left)Speaker 2 (right)

0:00 / 44.5 s

Our base model behaves more like the average person than an assistant. If you start talking about movies, it might start talking about food. With no reason to follow what the user says, it often refuses questions, claims to not know answers, or goes off on tangents. We post-trained the model to adjust this behavior.

Post-Training

In existing speech models, post-training is a tradeoff between desired behavior and naturalness. If you post-train too heavily, you end up with a model that has lost the human-like feel that it started with. If you don’t post-train enough, the model is unusable in any practical setting.

2.08 s · Prefix ends

0:00 / 29.28 s

We sought to strike a balance here, applying a light touch to our post-training and targeting specific, carefully-chosen behaviors using Reinforcement Learning. Our final RL pipeline consisted of only around 1,300 hours of rollouts.

RL reward
RL reward from step 0 to step 80, rising from −3.56 to −0.17.
Reward at each step over the course of our final RL pipeline.

In the early stages of RL, we played fixed prefixes against our model and rewarded on-topic and timely responses using GRPO. Once we saturated the reward against these fixed prefixes, we trained on a form of modified self-play, where our post-trained model speaks to another instance of the model.

This required us to build a good “user simulator” for our model to talk to. In text-based language models, user simulators are something that people have been researching for a long time. But there is nothing of the sort for full-duplex models. So we post-trained a version of our base model to adopt the user side of user-model interactions, giving our model a realistic conversation partner for RL rollouts. This allows us to better post-train on multi-speaker scenarios.

Multi-speaker RL reward
Multi-speaker RL reward over 6 consecutive runs and 103 epochs, with held-out reward as dots. Dashed lines divide the runs; training reward ends at −1.48.
Reward at each epoch across six consecutive RL runs on multi-speaker conversations, with held-out reward shown as dots. Dashed lines divide the runs, where scenario difficulty increases with each run, causing the sharp drops in reward.

We also trained checkpoints to be able to take in a text prefix, what we call a “reference document,” as additional input before a conversation. This allows us to provide the model with external information, hook it up to task-specific harnesses we’ve built, and compact conversations so that they can last longer than our native context window.

We encountered some surprising generalization results during reinforcement learning. During one training run, we performed self-play rollouts with our latest blog post as the reference document, on prefixes with the user debating the model about the blog. Despite only training on this one essay, this training generalized such that the model could debate competently about any article of similar length, with no leakage of information from our blog post:

Tanner (left)Will (right)

0:00 / 54.9 s

Results

Interact-0 is the only model that can behave naturally in group settings. Our checkpoint, named Orchestra, is trained to stay quiet most of the time, listening in on the conversation and chiming in when it has something useful to provide, or when it is called on.

Because no existing models can interact naturally in groups, there are no benchmarks for this. We created our own benchmarks to fill this need: Multi-Bench and Interruption-Bench.

Multi-Bench sets up scenarios where the model is in a conversation with two other people. The model must understand when the other people are talking to it, versus when they’re talking to each other, and respond or remain silent accordingly. In some scenarios, the model simply has to respond to its name; in others, it must decide based on context clues.

Interruption-Bench tracks the ability of a model to keep talking when the user says backchannels such as “uh-huh,” “yeah,” “go on,” during its response. When models are comfortable holding the floor, they allow users to freely express themselves during the conversation without worrying that the model will cut itself off.

Conversation benchmark results
ModelMulti-BenchHigher is betterInterruption-BenchHigher is better
Interact-074.2%65%
GPT Live 11.5%32%
Gemini 3.8 Live4.5%1%
Phonic 13.5%52%
Multi-Bench reports the share of episodes passed. Interruption-Bench reports the share of user backchannels the model kept talking through. Interact-0 is our Orchestra checkpoint on Multi-Bench and our Duet checkpoint on Interruption-Bench.
Model: LeilaSpeakStay quiet
JessMarco

Hobblings

We are hobbled by hockey and power outages. We trained Interact-0 anyway.

Our data guy sustained a concussion from a hockey game right before some of our most important post-training runs.

Hockey players making contact near the boards.

Our office suffered a sudden power outage last week, so we had to move our local Inference Server to our investors' office for the weekend.

Our local inference server in the back of a van.

References

  1. [1]Jin Xu et al. Qwen3-Omni Technical Report. arXiv:2509.17765, 2025.
  2. [2]Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for real-time dialogue. arXiv:2410.00037, 2024.