Sesame

Sesame AI: Crossing the Uncanny Valley of Conversational Voice

What is Sesame AI? Sesame AI is dedicated to achieving "voice presence" in artificial intelligence, aiming to make spoken interactions feel real, understood, and valued. Their research focuses on creating conversational partners that engage in genuine dialogue, building confidence and trust over time.

How does Sesame AI work? Sesame AI introduces the Conversational Speech Model (CSM), an end-to-end multimodal learning task using transformers. CSM leverages the history of the conversation to produce more natural and coherent speech.

Key Components:

• Emotional intelligence: reading and responding to emotional contexts.
• Conversational dynamics: natural timing, pauses, interruptions, and emphasis.
• Contextual awareness: adjusting tone and style to match the situation.
• Consistent personality: maintaining a coherent, reliable, and appropriate presence.

Technical Details of CSM:

• CSM operates as a single-stage model, improving efficiency and expressivity.
• It uses two autoregressive transformers based on the Llama architecture.
• The model processes interleaved text and audio to model the zeroth codebook.
• A separate audio decoder uses a distinct linear head for each codebook to reconstruct speech from the backbone’s representations.

Compute Amortization:

To address infrastructure challenges during training, Sesame AI uses a compute amortization scheme that alleviates the memory bottleneck while preserving the fidelity of the full RVQ codebooks. The audio decoder is trained on only a random 1/16 subset of the audio frames, while the zeroth codebook is trained on every frame.

Experiments and Results:

Sesame AI trained three model sizes (Tiny, Small, and Medium) on a large dataset of publicly available audio. Evaluation included objective metrics like Word Error Rate (WER) and Speaker Similarity (SIM), as well as new phonetic transcription-based benchmarks for homograph disambiguation and pronunciation consistency.

Subjective metrics, using Comparative Mean Opinion Score (CMOS) studies on the Expresso dataset, revealed that while naturalness is saturated, a gap remains between generated and human prosody in conversational speech generation.

Why choose Sesame AI? Sesame AI's approach offers a promising path toward more natural and engaging AI conversations. By focusing on emotional intelligence, contextual awareness, and conversational dynamics, Sesame AI aims to create digital companions that truly understand and respond to human needs.

How to use Sesame AI? Try the conversational speech preview on the Sesame AI website to experience the potential of their approach. The models will be available under an Apache 2.0 license.

Who is Sesame AI for? Sesame AI is for researchers, developers, and anyone interested in advancing the field of conversational AI. Their work has applications in various areas, including:

• AI assistants
• Customer service
• Education
• Entertainment

Open-Sourcing and Future Work:

Sesame AI is committed to open-sourcing key components of their research, enabling the community to experiment, build upon, and improve their approach. Future work includes scaling up model size, increasing dataset volume, expanding language support, and exploring ways to utilize pre-trained language models.