Meta’s new AI model is a real-time translation expert

Meta’s latest AI output is a major advancement for real-time text and speech translation.

On Tuesday, the company released SeamlessM4T: a multimodal model that translates text to speech and vice versa. Meta claims SeamlessM4T is “the first all-in-one multilingual multimodal AI translation and transcription model,” meaning it is uniquely able to translate and transcribe languages at the same time. SeamlessM4T can translate speech-to-text, speech-to-speech, text-to-speech, and text-to-text inputs for up to 100 languages. Translations for speech-to-speech and text-to-speech translations outputs support 35 languages.

Tweet may have been deleted

Like other AI models recently released by Meta, including Llama 2 and AudioCraft, SeamlessM4T is publicly available for researchers and developers with a research license. Alongside the model, Meta is also releasing its training dataset called SeamlessAlign, which has 270,000 hours of speech and text alignments. Unlike OpenAI and Google, Meta has made a point of making its models open-source and publicly available. Meta’s approach of launching open-source models has the dual effect of enabling developers to build and improve the products, while also winning points amongst AI ethicists who are calling for transparency of generative AI systems.

Meta’s open-source approach may seem altruistic, but it’s a strategic power move in a ruthlessly competitive market against other big tech companies developing AI. There’s also the issue of data collection that all AI models must contend with. According to the blog post, SeamlessM4T’s dataset (SeamlessAlign) consists of publicly available data, there are ethical and legal issues surrounding use of copyrighted works and personal data without consent.

Meta’s announcement didn’t detail specific plans for SeamlessM4T, only hinting that it wants “to explore how this foundational model can enable new communication capabilities.” In other words, we might someday see a consumer-facing version of SeamlessM4T on WhatsApp or Instagram.