What are the responsibilities and job description for the Azure Databricks Architect position at Reliable Software Resources?
Job Role: Azure Databricks Architect
Location: Remote and Travel
Hire-Type: Long-term
Key Responsibilities
Responsibilities below apply across the team. Individual engineers own the areas aligned to their role above.
- Security Platform — Pipeline Operations & Data Quality
- Security Platform — Customer Enablement
- Security Platform — New Source Onboarding
- Security Platform — Pipeline Development
- SOX Platform — Source Onboarding & Pipeline Management
- SOX Platform — Alert Management
- SOX Platform — Monitoring Team Support
- SOX Platform — Audit Support & Evidence
- SOX Platform — Application Development & Release Management
- Cross-Platform Responsibilities
Required Qualifications
Experience
- 5 years of data engineering experience with at least 2 years on Azure Databricks or equivalent cloud lakehouse platforms.
- Hands-on experience with Medallion Architecture pipelines, Delta Lake (MERGE, OPTIMIZE, VACUUM, Z-ORDER, schema enforcement), and Databricks Structured Streaming.
- Familiarity with SOX compliance, IT General Controls (ITGC), or regulated-environment data engineering; experience supporting audits or evidence collection.
Security Domain Knowledge
- Familiarity with OCSF (Open Cybersecurity Schema Framework) or willingness to develop deep expertise rapidly.
- Understanding of security log source types (EDR, firewall, IAM/PAM, CASB) and SOC/SIEM/SOAR workflows; awareness of MITRE ATT&CK framework.
Preferred Qualifications
- OCSF normalization implementation across multiple security log sources.
- Experience with Azure API Management (APIM) for data platform API governance.
- UEBA, threat scoring, or ML-backed security analytics (MLflow, Databricks Feature Store, Mosaic AI).
- SOX audit participation — auditor walkthroughs, evidence packages, IT General Controls testing.
- Certifications: Databricks Certified Data Engineer Associate/Professional, Azure Data Engineer Associate (DP-203), or equivalent.
- Graph-based correlation (Databricks GraphX); Delta Sharing.