Machine Learning Engineer / Senior Machine Learning Engineer

C3 AI Data Science team is dedicated to pushing the boundaries of what is possible with large-scale AI. We are seeking a hands-on Machine Learning Engineer to design, build, and operate abespoke, next-generation research platform dedicated to training novel, large-scale foundation models far beyond conventional LLM recipes.

This is a critical systems role. You will create the orchestration, secure data pathways, and frictionless developer experience that empowers our researchers to move fast, experiment securely, and scale complex training jobs on heterogeneous GPU clusters.

Responsibilities:

We are looking for an expert who can solve infrastructure problems where off-the-shelf cloud tools are insufficient.

• Design and manage the core research compute cluster, including node layouts, queues/partitions, preemption/fair-share policies, and multi-tenant isolation.
• Implement secure access controls for all users and services across the cluster using Kubernetes and/or SLURM.
• Build robust branch-to-experiment CI/CD workflows, encompassing templated job creation, config promotion, and integrated version control.
• Implement an experiment and metrics tracking system (runs, configs, checkpoints, logs) with searchable lineage to enable frictionless cross-team collaboration and sharing.
• Design and integrate auto-checkpointing, artifact retention, and necessary rollout/rollback mechanisms.
• Stand up robust dataset registries, ensuring data lineage, versioning, and secure access.
• Implement sharding, streaming, and prefetch mechanisms to support efficient TB-scale data corpora access and long-term archival with reproducible rehydration.
• Profile NCCL/I/O hotspots, optimize training throughput (mixed precision/AMP, ZeRO/FSDP, kernel fusion, caching).
• Harden training pipelines for scale and resilience, including checkpoint recovery, and tolerance for spot/preemptible instances.
• Build opinionated templates, job specifications, and guardrails to ensure researchers can focus on modifying custom training code and recipes without fighting infrastructure bottlenecks.

Qualifications:

• BS/MS in Computer Science/Electrical Engineering or equivalent deep, practical experience.
• 5 years of work experience (8 years for Senior Machine Learning Engineer)
• Proven track record building custom ML/HPC platforms for specialized research (e.g., novel model architectures, time-series, multimodal AI) where commercial cloud tools were insufficient.
• Deep expertise with Kubernetes and/or SLURM on GPU clusters, including proficiency with containers, images, and multi-node scheduling.
• Strong software development skills in Python and one of Go, C , or Rust. Comfortable developing controllers/operators, high-performance services, and CLI tooling on Linux.
• Practical, hands-on knowledge of distributed ML frameworks (PyTorch DDP/FSDP/ZeRO, DeepSpeed, or JAX/TPU) and performance profiling (NCCL, CUDA basics, I/O performance).
• Experience with object stores, Parquet format, dataset version control, streaming/sharding techniques, and efficient artifact management for checkpoints and logs.
• Practical experience with observability (Prometheus/Grafana/OpenTelemetry) and infra-as-code (Terraform/Helm/Ansible).

Preferred Qualifications:

• Experience with high-speed networking and storage, including InfiniBand/RDMA, GPUDirect-RDMA, NVLink topology, and high-throughput file/object systems.
• Direct experience modifying or working with K8s device plugins, custom schedulers/quotas, or SLURM internals (fair-share/preemption).
• Expertise in implementing true reproducibility at scale: seeding, deterministic builds, environment capture, and building robust dataset & experiment lineage that guarantee re-runnability months later.
• Experience with advanced performance work such as kernel fusion, custom CUDA operations, and fine-tuning complex FSDP/ZeRO configurations.
• A pragmatic, product-focused approach to researcher ergonomics, demonstrated by platforms you have shipped that materially increased experiment throughput and velocity.