Senior Site Reliability Engineer (Seed Model - Infrastructures - SRE)

About the Team The Seed Infrastructures team oversees the distributed training, reinforcement learning framework, high-performance inference, and heterogeneous hardware compilation technologies for AI foundation models. The Seed Infra SRE team is responsible for the delivery and stability of the compute infrastructure behind Seed's foundation models. We own the reliability of large-scale heterogeneous GPU clusters, the training and inference platforms that run on them, and the capacity, network and storage systems that keep them running. Our work directly determines how fast and how reliably we can train and serve the models. Responsibilities - Own the stability of training systems and heterogeneous inference systems for large-scale generative AI workloads (LLM, text-to-image, text-to-video, multimodal understanding); drive deep performance optimization and resolve core issues such as training interruptions, inference latency and throughput bottlenecks. - Design and iterate a flexible, efficient resource scheduling system; improve scheduling efficiency, resource utilization and scheduling explainability; support fine-grained scheduling and priority management across workload types (training, offline inference, online serving). - Lead the planning, deployment, operation and optimization of large-scale high-speed networks (RDMA) and heterogeneous compute clusters (GPU / NPU / XPU); ensure sound cluster topology, stable network bandwidth and efficient delivery of compute resources. - Own the architecture, selection and operation of high-performance storage systems; ensure storage I/O performance, data reliability and stability to meet the data access needs of large-scale training and inference. - Own end-to-end quality of capacity delivery; build capacity planning methodology, and combine business growth forecasts to complete resource expansion and configuration optimization so that capacity is delivered on time, reliably and cost-effectively. - Drive the build-out of a unified compute pool with co-located training and inference workloads; improve compute utilization for offline inference and similar scenarios; establish an operating framework for compute efficiency and cost control, and codify reusable methodology. - Safeguard the stability of complex online services and engineering architecture; build monitoring, alerting and automated remediation that cover the full training and inference path; respond to and resolve complex production issues quickly. - Partner with algorithm, product and operations teams to deliver MLOps technical solutions; establish engineering best practices for large-scale generative AI, and raise the technical bar of the team.

Qualifications

Minimum Qualifications - Bachelor's degree or above in Computer Science, Software Engineering or a related field, with 3 years of experience in large-scale machine learning / generative AI MLOps or closely related infrastructure work. - Proficient in Python and/or Go; solid Linux development and operations skills; expert in shell scripting and able to independently build automation tooling. - Expert in containerization (Docker / Kata) and orchestration (Kubernetes), with hands-on experience deploying, operating and optimizing large-scale heterogeneous compute clusters (GPU / NPU). - Familiar with the principles and operation of large-scale high-speed networking (RDMA) and high-performance storage (distributed / object storage), with practical experience optimizing these systems. - Expert in the design and optimization of resource scheduling systems, with hands-on experience iterating scheduling policies and improving utilization; understands the key design considerations for scheduling explainability. - Experience ensuring stability and optimizing performance of large-scale generative AI (LLM / image / video generation) training or inference systems; able to quickly localize and resolve complex issues across the full training and inference path. - Project experience with co-located training / inference compute pools or offline inference efficiency; understands the core metrics and levers of compute operations. - Strong communication, cross-team collaboration and self-drive; a strong sense of ownership; able to operate under the pressure of large-scale production environments. Preferred Qualifications - End-to-end experience productionizing large-scale generative AI models (LLM, text-to-image, text-to-video). - Experience managing heterogeneous compute clusters and capacity planning in multi-cloud or hybrid-cloud environments. - Familiarity with distributed training frameworks and inference optimization tooling, with hands-on performance tuning experience. - Experience with large-scale network performance tuning (RDMA bandwidth / latency) or breaking through storage I/O bottlenecks. - Experience delivering large and complex compute capacity and building the associated operating framework, with quantified results in cost control or utilization improvement. - Experience designing large-scale distributed system architectures and running fault-injection / chaos exercises.