Principal Cloud Data Architect (Databricks & AWS Platform)

Job title: Principal Cloud Data Architect (Databricks & AWS Platform)

Location: New Jersey

Experience Level: 10 years (Architect Level)

Position Overview

We are seeking a highly experienced, Architect-Level Databricks & AWS Platform Specialist to lead our platform support, security governance, and capability enhancement initiatives. In this role, you will be responsible for the architectural integrity, optimization, and day-to-day operational excellence of our enterprise Databricks environment hosted on AWS.

The ideal candidate will combine deep technical expertise in data engineering infrastructure with a strategic mindset to enhance platform capabilities, mentor engineering teams, and ensure robust, production-grade stability.

Key Responsibilities

Platform Architecture & Capability Enhancement

• Strategic Roadmap: Stay updated on emerging Databricks and AWS developments, providing strategic recommendations, proof-of-concepts, and roadmaps for platform enhancements and new features.
• Feature Deployment: Lead the release management, deployment, and configuration of new Databricks features and enterprise capabilities, ensuring strict alignment with security and performance best practices.
• Enablement & Training: Conduct advanced training sessions and architect comprehensive technical documentation to empower data engineering and data science teams to leverage Databricks effectively.

AWS Infrastructure & Security Governance

• Cloud Resource Management: Architect and manage AWS resources tightly integrated with Databricks, including IAM roles, policies, security groups, VPC peering, PrivateLink, and networking configurations.
• Security & Compliance: Ensure data governance and platform security align with enterprise standards (e.g., Unity Catalog implementation, encryption, and network isolation).

Operations, Optimization & Support

• Performance Engineering: Collaborate closely with data engineering teams to diagnose, optimize, and fine-tune complex data pipelines, Spark workflows, and SQL warehouses on Databricks.
• Tier-3 Technical Support: Provide high-level technical support for the Databricks platform, resolving deep architectural bottlenecks, connectivity issues, and performance anomalies.
• Monitoring & Observability: Design and maintain robust monitoring, alerting, and troubleshooting frameworks for Databricks jobs, clusters, and notebooks to ensure maximum operational efficiency and cost optimization.
• Incident Response: Participate in an architectural on-call escalation rotation to respond to urgent platform incidents and ensure high availability.

Required Qualifications & Skills

• Core Expertise: Demonstrated experience as an Architect or Principal Engineer managing enterprise-scale Databricks environments natively on AWS.
• AWS Mastery: Deep understanding of AWS infrastructure, specifically advanced networking (VPCs, Route 53, Security Groups) and enterprise security (IAM, KMS, Cross-account roles).
• Data Engineering Foundation: Strong background in Apache Spark architecture, data pipelining (Delta Lake), optimization techniques (caching, partitioning, Z-Ordering), and languages like PySpark, Scala, or SQL.
• DevOps/IaC: Experience with Infrastructure as Code (e.g., Terraform) for deploying Databricks workspaces and AWS resources is highly preferred.
• Soft Skills: Exceptional communication skills with the ability to bridge the gap between deep technical implementation and high-level stakeholder strategy.