What are the responsibilities and job description for the Databricks Architect position at E-Solutions?
Job Title: Databricks Architect
Location: Jersey City, NJ (3–4 Days Onsite)
Employment Type: W2 Contract Only
Project: Lakehouse Modernization Program
Role Overview
We are seeking a senior Databricks Architect to lead large-scale enterprise data platform modernization initiatives. The role focuses on migrating legacy and on-premises data workloads to cloud-native Databricks Lakehouse architectures while ensuring governance, security, resiliency, and regulatory compliance.
Key Responsibilities
Cloud Migration & Modernization
- Lead migration of legacy/on-prem data platforms to Databricks on cloud.
- Perform application, data, and pipeline modernization using cloud-native Databricks patterns.
- Drive migration planning, dependency analysis, sequencing, and cutover strategies.
- Support coexistence models and transition from dual-run to cloud-only environments.
Databricks Lakehouse Engineering
- Design and implement Databricks Lakehouse architectures (Bronze, Silver, Gold layers).
- Develop scalable batch and streaming data pipelines using PySpark and Spark SQL.
- Utilize Delta Lake for reliability, versioning, and performance optimization.
- Optimize cloud resource utilization and cost efficiency.
Data Governance & Controls
- Implement data quality, reconciliation, and completeness controls.
- Ensure compliance with governance, audit, and regulatory requirements.
- Maintain data lineage, traceability, and explainability across data assets.
- Support critical reporting, finance, operations, and reconciliation workloads.
Security & Resilience
- Implement RBAC, identity management, and secure access controls.
- Enforce data encryption, masking, and classification standards.
- Ensure operational resilience, disaster recovery, and compliance readiness.
- Collaborate with cloud and security teams for approvals and certifications.
Analytics & AI Enablement
- Enable downstream BI, reporting, and analytics workloads.
- Prepare data platforms for AI/ML and GenAI use cases.
- Support enterprise reporting and management information initiatives.
Required Qualifications
- 8–12 years of experience in Data Engineering, Data Architecture, or Platform Modernization.
- Strong hands-on experience with Databricks in large-scale enterprise environments.
- Proven experience leading on-premises to cloud migration programs.
- Deep expertise in Apache Spark, PySpark, and Spark SQL.
- Experience implementing data quality, governance, and reconciliation controls.
- Experience working in regulated environments (Banking/Financial Services preferred).
Preferred Qualifications
- Experience with large enterprise cloud transformation programs.
- Knowledge of legacy data platform modernization strategies.
- Experience supporting Finance, Operations, Reconciliation, or Reporting domains.
- Exposure to MLflow, AI/ML pipelines, or GenAI solutions.
- Strong understanding of enterprise transformation and modernization initiatives.
Required Skills
- Databricks Lakehouse Architecture
- Apache Spark
- PySpark
- Spark SQL
- Delta Lake
- Cloud Migration
- Data Governance
- Data Quality & Reconciliation
- RBAC & Security Controls
- Enterprise Data Architecture