Research Scientist - Omni LLM

Team: AI Research, reporting to AI Director / Head of Research (Onsite/Remote)

Focus: Speech & Multimodal AI / Omni / Any-to-Any Foundation Models

About Propio

Propio Language Services is one of the top 5 providers of high-quality, real-time multilingual interpretation, translation, and localization services, operating at 9-figure scale across healthcare, legal, and other industries. We are driven by a passion for cutting-edge technology and exceptional service, building seamless experiences that bridge communication gaps across languages, cultures, and modalities.

The Role

We build real-time speech translation and interpretation technology, and we train our own models to power it. We’re looking for an exceptional research scientist to build our omni foundation model: a single model that understands and generates across speech, text, and vision.

Our north star is end-to-end, voice- and prosody-preserving, low-latency speech-to-speech generative model. This is fundamentally a unified multimodal modeling problem. Speech is our anchor modality, and the hardest open questions sit where omni modeling meets real-time interaction: cross-modal tokenization, streaming low-latency generation, and any-to-any generation quality. You’ll help define how we get there.

Key Responsibility - What You’ll Do

• Own the research roadmap for our omni model, from architecture selection through pretraining and post-training.
• Design the unified architecture. Build and optimize unified network architectures that fuse different tokenizers and encoders, such as Vision Transformers for images, audio encoders and neural codecs for speech, and sequence or time-series models for streaming signals. Own the core modeling paradigm (autoregressive over a shared discrete vocabulary, masked discrete diffusion, or hybrid AR diffusion/flow), unifying understanding and generation in one model.
• Design cross-modal tokenization. For speech, this means choosing semantic vs. acoustic tokens with streaming low-latency generation in mind and avoiding the “complete-then-decode” bottleneck. For vision, it means selecting and adapting image tokenizers for both understanding and generation.
• Solve modality interference via modality-specific routing / MoE, and advance cross-modal alignment and interleaved multimodal modeling.
• Drive high-quality any-to-any generation (speech and image), not just any-to-text understanding.
• Lead pretraining and post-training (SFT, RLHF/RLVR), including streaming and full-duplex interaction capabilities.
• Build large-scale multimodal data strategies, including paired, interleaved, and synthetic data.
• Own evaluation across understanding, generation quality, and latency, partnering with professional interpreters and linguists.
• Set technical direction and make pragmatic, evidence-driven choices in collaboration with the data, speech, MT, and real-time-systems teams.

Qualifications

• Ph.D. in Computer Science, Machine Learning, Speech Processing, NLP, Computer Vision, or a related field, or equivalent industry research experience.
• 3 years of research experience with deep, demonstrated expertise in multimodal AI or foundation models.
• Track record of first-author publications at top venues (NeurIPS, ICML, ICLR, CVPR, ICCV, ACL, EMNLP, Interspeech, ICASSP).
• Hands-on experience training modern LLMs / foundation models at scale (pretraining and/or post-training, RLHF).
• Strong understanding of multimodal tokenization and generative modeling, such as VQ/RVQ tokenizers, neural audio codecs, autoregressive token LMs, and diffusion/flow-based models.
• Experience owning the end-to-end research lifecycle, including problem formulation, baseline reproduction, dataset and model design, training-objective development, large-scale experimentation, ablation and error analysis, rigorous evaluation, and production handoff.
• Strong engineering in Python PyTorch, with the ability to design and run experiments at scale using distributed training (e.g., FSDP, DeepSpeed, Megatron).

Preferred Qualifications

• Experience building omni models (GPT-4o-style) or work in the lineage of Chameleon, AnyGPT, NExT-GPT, Qwen-Omni, Moshi, Mini-Omni, Emu3, Janus, Show-o, or Transfusion.
• Speech tokenization, streaming speech generation, and full-duplex spoken dialogue.
• MoE architectures, modality routers, or Mixture-of-Transformers; discrete-diffusion multimodal generation.
• Large-scale multimodal data construction and curation.
• Familiarity with speech translation and related tools (Whisper, wav2vec/HuBERT, SeamlessM4T) and simultaneous policies (wait-k, AlignAtt, MMA, Local Agreement).
• Experience shipping production real-time multimodal systems or optimizing low-latency inference.
• Low-resource / massively multilingual modeling, open-source contributions, or agentic / tool-use modeling.