Sr. Databricks Solutions Architect

Note: Only send your TOP 1 candidate. Candidates should have some kind of Healthcare background. When you submit a candidate, they must have an active LinkedIn page with a photo, and 2 managerial references included.Job Title: Sr. Databricks Solutions ArchitectJob Location: Must be able to come to the office when needed - Washington, DC (Candidate must be in the DMV area)Length: Contract to HireStart Date: January 19th or January 12Pay Rate: $60/hour C2CConversion Salary: Up to $170KBackground: Standard background check requiredWork Auth: US Citizen, Green Card ONLYTop Skills Needed:Deep hands-on expertise with Databricks platform architecture and governanceUnity Catalog, workspaces, external locations, compute, access controls, cluster governance.Reliability engineering, monitoring, and operational hardening of the LakehouseObservability, alerting, DR readiness, backup/restore, performance tuning, incident response.Strong experience with ADF, CI/CD, and Terraform for orchestrating and managing the LakehousePipeline orchestration, IaC, DevOps, environment promotion, compute policies. Typical Day-to-Day:Design how the Databricks Lakehouse should work including the structure, tools, standards, and best practicesGuide engineering teams on how to build pipelines and use Databricks correctlySolve technical issues when data jobs fail or performance slowsWork with stakeholders to understand data needs and deliver solutionsSet standards for security, governance, naming conventions, and architectureEnsure the Databricks platform is stable, reliable, and always availableBuild and maintain monitoring, alerting, logging, and health dashboardsStrengthen and fix ingestion pipelines (ADF ? landing ? raw ? curated)Improve data quality checks, anomaly detection, and pipeline reliabilityManage CI/CD pipelines and deployment processes using Azure DevOps or GitHubUse Terraform (IaC) to deploy and manage Databricks and Azure infrastructurePartner with Security and FinOps on access controls, compliance, and cost governanceMentor the Data Engineer and support distributed data engineering teams across the organization Key Responsibilities1. Lakehouse Architecture & Platform Administration(Approximately 60% of role when combined with mentoring & code review)Serve as the primary architect and administrator for the Azure Databricks Lakehouse (Unity Catalog, workspaces, external locations, compute, access controls).Lead execution of a Minimal Viable Hardening Roadmap for the platform, prioritizing:High availability and DR readinessBackup/restore patterns for data and metadataPlatform observability and operational metricsSecure and maintainable catalog/namespace structureRobust and proactive data quality assuranceImplement and evolve naming conventions, environment strategies, and platform standards that enable long-term maintainability and safe scaling.Act as the Lakehouse-facing counterpart to Enterprise Architecture and Security, collaborating on network architecture, identity & access, compliance controls, and platform guardrails. 2. Reliability, Monitoring, and Incident ManagementDesign, implement, and maintain comprehensive monitoring and alerting for Lakehouse platform components, ingestion jobs, key data assets, and system health indicators.Oversee end-to-end reliability engineering, including capacity planning, throughput tuning, job performance optimization, and preventative maintenance (e.g., IR updates, compute policy reviews).Participate in — and help shape — the on-call rotation for high-priority incidents affecting production workloads, including rapid diagnosis and mitigation during off-hours as needed.Develop and maintain incident response runbooks, escalation pathways, stakeholder communication protocols, and operational readiness checklists.Lead or participate in post-incident Root Cause Analyses, ensuring durable remediation and preventing recurrence.Conduct periodic DR and failover simulations, validating RPO/RTO and documenting improvements.This role is foundational to ensuring 24/7/365 availability and timely delivery of mission-critical data for clinical, financial, operational, and analytical needs. 3. Pipeline Reliability, Ingestion Patterns & Data QualityStrengthen and standardize ingestion pipelines (ADF ? landing ? raw ? curated), including watermarking, incremental logic, backfills, and retry/cancel/resume patterns.Collaborate with the Data Engineer to modernize logging, automated anomaly detection, pipeline health dashboards, and DQ validation automation.Provide architectural guidance, code reviews, mentoring, and best-practice patterns to distributed engineering teams across MedStar.Support stabilization of existing ingestion and transformation pipelines across clinical (notes, OHDSI), financial, operational, and quality use cases. 4. DevOps, CI/CD, and Infrastructure as CodeAdminister and improve CI/CD pipelines using Azure DevOps or GitHub Enterprise.Support automated testing, environment promotion, and rollback patterns for Databricks and dbt assets.Maintain and extend Terraform (or adopt Terraform from another IaC background) for Databricks, storage, networking, compute policies, and related infrastructure.Promote version control standards, branching strategies, and deployment governance across data engineering teams. 5. Security, FinOps, and Guardrails PartnershipPartner with Enterprise Architecture and Security on platform access controls, identity strategy, encryption, networking, and compliance.Implement and enforce cost tagging, compute policies, and alerts supporting FinOps transparency and cost governance.Collaborate with the team defining agentic coding guardrails, ensuring the Lakehouse platform supports safe & compliant use of AI-assisted code generation and execution.Help assess and optimize serverless SQL, serverless Python, and job compute patterns for cost-efficiency and reliability. 6. Mentorship, Collaboration, & Distributed EnablementMentor the mid-level Data Engineer on Databricks, ADF, dbt, observability, DevOps, Terraform, and operational engineering patterns.Provide guidance, design patterns, and code review support to multiple distributed data engineering teams (Finance, MCPI, Safety/Risk, Quality, Digital Transformation, etc.).Lead platform knowledge-sharing efforts through documentation, workshops, and best-practice guidance.Demonstrate strong collaboration skills, balancing independence with alignment across teams. 7. Optional / Nice-to-Have: OHDSI Platform Support(Not required for hiring; can be learned on the job.)Assist with or support operational administration of the OHDSI/OMOP stack (Atlas, WebAPI, vocabularies, Kubernetes deployments).Collaborate with partners to ensure the OHDSI platform is secure, maintainable, and well-integrated with the Lakehouse. Required Qualifications5 years in cloud data engineering, platform engineering, or solution architecture.Strong hands-on expertise in Azure Databricks:Unity CatalogWorkspaces & external locationsSQL/Python notebooks & JobsCluster/warehouse governanceSolid working experience with Azure Data Factory (pipelines, IRs, linked services).Strong SQL and Python engineering skills.Experience with CI/CD in Azure DevOps or GitHub Enterprise.Experience with Terraform or another IaC framework, and willingness to adopt Terraform.Demonstrated ability to design or support monitoring, alerting, logging, or reliability systems.Strong communication, collaboration, and problem-solving skills. Preferred Qualifications (Optional)Advanced Terraform experience.Familiarity with healthcare, HIPAA, PHI, or regulated environments.Experience with Purview or enterprise cataloging.Exposure to OHDSI/OMOP.Experience optimizing or refactoring legacy ingestion pipelines.Experience supporting secure, controlled AI/agentic execution environments.Experience with EPIC EHR data exchange and/or EPIC Caboodle or Cogito analytics suite. Personal AttributesHands-on, pragmatic, and operationally minded.Comfortable leading both architecture and implementation.Collaborative and mentorship-oriented; thrives in small core teams with broad influence.Values platform stability, observability, and hardening over shiny features.Curious and adaptable, especially with emerging AI-assisted engineering patterns.Ability to remain calm and effective during incidents and high-pressure situations

Salary : $170,000