Databricks Engineer

Job Role : Databricks Engineer

Location : Maryland

Client : University of Maryland Global Campus

We are seeking a Databricks Engineer to design, build, and operate a Data & AI platform with a strong

foundation in the Medallion Architecture (raw/bronze, curated/silver, and mart/gold layers). This

platform will orchestrate complex data workflows and scalable ELT pipelines to integrate data from

enterprise systems such as PeopleSoft, D2L, and Salesforce, delivering high-quality, governed data

for machine learning, AI/BI, and analytics at scale.

You will play a critical role in engineering the infrastructure and workflows that enable seamless data

flow across the enterprise, ensure operational excellence, and provide the backbone for strategic

decision-making, predictive modeling, and innovation.

Responsibilities

• Data & AI Platform Engineering (Databricks-Centric):
• Design, implement, and optimize end-to-end data pipelines on Databricks, following the
  
  

Medallion Architecture principles.

• Build robust and scalable ETL/ELT pipelines using Apache Spark and Delta Lake to transform
  
  

raw (bronze) data into trusted curated (silver) and analytics-ready (gold) data layers.

• Operationalize Databricks Workflows for orchestration, dependency management, and
  
  

pipeline automation.

• Apply schema evolution and data versioning to support agile data development.
• Platform Integration & Data Ingestion:
• Connect and ingest data from enterprise systems such as PeopleSoft, D2L, and Salesforce using
  
  

APIs, JDBC, or other integration frameworks.

• Implement connectors and ingestion frameworks that accommodate structured, semi
  
  

structured, and unstructured data.

• Design standardized data ingestion processes with automated error handling, retries, and
  
  

alerting.

• Data Quality, Monitoring, and Governance:
• Develop data quality checks, validation rules, and anomaly detection mechanisms to ensure
  
  

data integrity across all layers.

• Integrate monitoring and observability tools (e.g., Databricks metrics, Grafana) to track ETL
  
  

performance, latency, and failures.

• Implement Unity Catalog or equivalent tools for centralized metadata management, data
  
  

lineage, and governance policy enforcement.

• Security, Privacy, and Compliance:
• Enforce data security best practices including row-level security, encryption at rest/in transit,
  
  

and fine-grained access control via Unity Catalog.

• Design and implement data masking, tokenization, and anonymization for compliance with
  
  

privacy regulations (e.g., GDPR, FERPA).

• Work with security teams to audit and certify compliance controls.