What are the responsibilities and job description for the Lead Databricks Architect @ NYC, NY (Day 1 Onsite) position at Kaizen Technologies?
Title: Data Architect with Databricks
Location: NYC, NY – Day 1 Onsite
Primary Skill: Data Bricks, PySpark
Certification with Databricks is mandatory
Job Description:
- Hands-on Data Architect (Databricks Lakehouse)
Role Overview:
- Own the target-state architecture in Databricks and actively contribute to pipeline development.
- Define how API/FTP data flows into Bronze/Silver/Gold layers with cleansing and enrichment logic.
Key Responsibilities:
- Design Databricks Lakehouse architecture (Bronze/Silver/Gold)
- Define ingestion patterns for API & FTP
- Architect scalable cleansing & enrichment frameworks
- Translate legacy SQL logic into Spark-based transformations
- Define Delta Lake optimization strategy
- Establish security, governance, and PHI controls
- Implement CI/CD for data pipelines
- Mentor engineering team
Required Skills
- Strong expertise in Databricks & Delta Lake
- Advanced PySpark & Spark SQL
- Experience designing data pipelines from scratch
- Strong understanding of SQL Server logic & stored procedures
- Azure cloud experience
- Healthcare data architecture experience preferred