What are the responsibilities and job description for the Databricks Platform Engineer position at InnoCore Solutions, Inc.?
The Databricks Platform Engineer is responsible for the administration, governance, and operational management of the enterprise Databricks platform on AWS. This role serves as the primary liaison between platform engineering and internal customers, ensuring the platform is secure, scalable, cost-efficient, and aligned with business and data strategy objectives.
Responsibilities:
- Serve as the primary point of contact and liaison for internal customers, gathering requirements and providing guidance on Databricks platform capabilities and best practices.
- Design, develop, and maintain the AWS infrastructure supporting the Databricks platform, ensuring high availability, security, and scalability.
- Drive Databricks platform architecture, establish technical standards, and define the platform feature roadmap and adoption strategy.
- Provision and administer Databricks workspaces, including workspace configuration, metadata management, and environment setup.
- Implement and manage Unity Catalog, including catalogs, schemas, and data governance policies.
- Administer identity and access management by creating and managing Active Directory and cloud identity groups, roles, and permissions.
- Establish and maintain data governance frameworks, including access controls, data classification, and compliance requirements.
- Manage onboarding and offboarding processes for product teams and users, ensuring appropriate access provisioning and deprovisioning.
- Monitor platform operations, performance, availability, and reliability, and proactively resolve issues impacting platform users.
- Develop and maintain platform monitoring, utilization dashboards, and cost reporting to optimize cloud and Databricks resource consumption.
- Analyze platform usage patterns and recommend strategies to improve performance, scalability, and cost efficiency.
- Lead incident management activities, coordinate issue resolution, and perform root cause analysis for production incidents.
- Create and maintain operational documentation, governance procedures, and platform standards.
- Collaborate with data engineering, analytics, security, and infrastructure teams to continuously improve the Databricks platform and user experience.
Requirements:
- 2 years of experience in Cloud Engineering, Data Platform Engineering, or Platform Administration roles.
- 2 years of hands-on experience administering Databricks platforms in AWS environments.
- Strong experience with AWS services, including VPC, IAM, EC2, S3, CloudWatch, and networking concepts.
- Experience implementing and administering Databricks Unity Catalog, catalogs, schemas, and access controls.
- Experience with identity and access management, including Microsoft Active Directory, Microsoft Entra ID, and role-based access control (RBAC).
- Strong understanding of data governance, security, and compliance principles.
- Experience with infrastructure automation and Infrastructure-as-Code tools such as Terraform or CloudFormation.
- Experience monitoring cloud costs and implementing platform utilization and cost optimization strategies.
- Experience supporting platform operations, incident management, and production support processes.
- Strong scripting skills using Python, Bash, or PowerShell.
- Excellent analytical, problem-solving, communication, and stakeholder management skills.
- Experience working in Agile and DevOps environments.
- Databricks Certified Platform Administrator certification.
Education:
- Bachelor's degree in Computer Science/Engineering or a related field