What are the responsibilities and job description for the Senior Databricks Platform Engineer (AWS) position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, NGTalentTech Group LLC, is seeking the following. Apply via Dice today!
Responsibilities
Responsibilities
- Environment Provisioning & Configuration
- Set up Databricks workspaces in designated AWS regions.
- Configure networking (VPC, subnets, security groups) and establish secure connectivity.
- Implement storage and credential configurations, including KMS encryption keys for data security.
- Access Management & Security
- Address WIZ Violations
- Define and implement role-based access controls (RBAC) using Unity Catalog.
- Integrate Databricks with Vanguard’s Identity Provider (SSO) and manage authentication flows.
- Ensure compliance with security standards and governance controls.
- Platform Integration & Data Access
- Validate external data access strategies and configure connectors for S3 and other data sources.
- Enable ingestion frameworks and orchestration of notebooks for data pipelines.
- Operational Enablement
- Support CI/CD deployment pipelines for Databricks notebooks and jobs.
- Implement monitoring for cluster lifecycle, cost optimization, and performance tuning.
- Assist with troubleshooting and error handling during execution.
- Compliance & Governance
- Apply data governance and ML/GenAI governance controls as per enterprise standards.
- Ensure audit logging and lineage tracking for all data assets.
- Collaboration & Documentation
- Work with internal teams (EADP, CTO, ES&F) to align with architecture strategy and best practices.
- Document configurations, processes, and operational runbooks for future reference.
- Strong Databricks Platform knowledge with AWS, Proficiency in AWS services such as CFT, EC2, S3, Lambda, and RDS, along with the AWS SDK and CLI and Terraform.
- Familiarity with AWS infrastructure as code tools like CloudFormation for automating cloud deployments.
- Understanding of serverless architecture and microservices, with hands-on experience in implementing scalable AWS solutions.
- Should have grasp of security best practices within AWS, including IAM roles, VPCs, and encryption mechanisms.
- Strong analytical and problem-solving skills, essential for debugging and optimizing AWS resource utilization.
- Good communication skills, enabling clear and effective interaction with both technical and non-technical team members and stakeholders.