Lead Data Platform Engineer

Job Title: Lead Data Platform Engineer

Location: Irvine, CA (Onsite)

Employment Type: W2 Only

Job Summary

We are seeking an experienced Lead Data Platform Engineer to design, build, and manage enterprise-scale cloud-native data platforms. The ideal candidate will have extensive experience with Databricks, AWS, Terraform, DevOps, and Platform Engineering, along with a strong background in building secure, scalable, and highly available data platforms. This role requires close collaboration with development, infrastructure, security, and operations teams to deliver reliable and efficient data platform solutions.

Key Responsibilities

• Design, build, and manage enterprise-scale cloud-native data platforms.
• Lead Databricks platform administration, engineering, and operational activities.
• Design and implement Lakehouse Architecture using Delta Lake.
• Manage Databricks Workflows, Unity Catalog, security configurations, and access controls.
• Optimize platform performance, scalability, availability, and cost efficiency.
• Implement governance controls across development, testing, and production environments.
• Design and deploy AWS-based infrastructure using services such as S3, Glue, Lambda, ECS, EKS, CloudWatch, IAM, Secrets Manager, and VPC.
• Develop Infrastructure as Code (IaC) solutions using Terraform.
• Create reusable Terraform modules for infrastructure provisioning.
• Design, build, and maintain CI/CD pipelines for automated application and infrastructure deployments.
• Improve deployment automation and platform reliability using DevOps best practices.
• Utilize GitHub Actions, GitLab CI/CD, Azure DevOps, Jenkins, Docker, and Kubernetes.
• Implement monitoring, observability, logging, and incident management processes.
• Perform root cause analysis and implement preventive measures to improve platform stability.
• Collaborate with engineering, infrastructure, security, and operations teams to deliver secure, scalable, and highly available platform solutions.
• Ensure compliance with enterprise security, governance, and operational standards.
• Provide technical leadership, best practices, and mentoring to engineering teams.

Required Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field.
• 10 years of experience in Platform Engineering, Cloud Engineering, Data Engineering, or a related field.
• Hands-on experience designing, implementing, and supporting enterprise-scale cloud-native data platforms.
• Strong expertise in Databricks platform administration and Lakehouse Architecture.
• Extensive experience with AWS cloud infrastructure and Infrastructure as Code (Terraform).
• Strong understanding of DevOps, Cloud Security, Infrastructure Automation, and Platform Operations.
• Experience with containerization technologies such as Docker and Kubernetes.
• Strong scripting/programming experience using Python.
• Excellent analytical, troubleshooting, communication, and collaboration skills.

Required Skills

Data Platform

• Databricks Administration
• Lakehouse Architecture
• Delta Lake
• Unity Catalog
• dbt

Cloud & Infrastructure

• AWS Cloud Services
• Terraform
• Infrastructure as Code (IaC)

DevOps & CI/CD

• GitHub Actions
• GitLab CI/CD
• Azure DevOps
• Jenkins
• Docker
• Kubernetes
• CI/CD Pipelines
• DevOps

Programming

• Python

Platform Operations

• Monitoring & Observability
• Security & Access Management
• Platform Engineering
• Cloud Engineering

Preferred Qualifications

• Experience building highly available enterprise data platforms.
• Experience with cloud governance, security, and compliance.
• Strong understanding of platform reliability engineering (SRE) principles.
• Experience supporting large-scale data analytics and AI/ML platforms.
• Ability to lead platform modernization and infrastructure automation initiatives