What are the responsibilities and job description for the Data Engineer - Databricks position at System One?
Data Engineer – Azure Databricks
Contract-to-Hire
Pittsburgh, PA – Onsite
Job ID J0726-0569
Visa : USC, GC, EAD (No Sponsorship)
Position Overview
- Seeking an experienced Data Engineer to join a high-performing data engineering team responsible for building modern, cloud-native data platforms on Microsoft Azure.
- This role will focus on designing and implementing scalable, secure, and high-performance data solutions using Azure Databricks, Apache Spark, PySpark, SQL, Delta Lake, and Unity Catalog. The position requires strong expertise in enterprise data governance, metadata management, security, and access control within the Investment Banking, Securities, or Capital Markets domain.
- The ideal candidate is passionate about cloud data engineering, enjoys solving complex data challenges, and thrives in a collaborative Agile environment.
Required Competencies
| Competency | Importance | Proficiency Level |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Design, develop, and maintain scalable data pipelines using Azure Databricks.
- Build and optimize ETL and ELT workflows using PySpark, SQL, Apache Spark, and Delta Lake.
- Implement and manage Unity Catalog for enterprise data governance, metadata management, security, and fine-grained access control.
- Develop cloud-native data engineering solutions on the Microsoft Azure platform.
- Collaborate with architects, business analysts, and cross-functional teams to translate business requirements into technical solutions.
- Optimize Spark jobs and data pipelines for performance, scalability, and cost efficiency.
- Ensure data quality, reliability, governance, and regulatory compliance through validation, monitoring, and engineering best practices.
- Support production deployments, troubleshooting, root-cause analysis, and performance tuning.
- Participate in code reviews, technical documentation, and continuous improvement initiatives.
- Apply cloud data engineering, security, CI/CD, and DevOps best practices within Azure environments.
- 8+ years of overall data engineering or related technology experience preferred.
- Minimum 5 years of hands-on Azure Databricks experience.
- Mandatory hands-on experience implementing and managing Unity Catalog.
- Expert-level experience with Apache Spark, PySpark, and SQL.
- Strong experience developing enterprise-scale data engineering solutions on Microsoft Azure.
- Strong experience with Delta Lake, ETL/ELT development, and data pipeline implementation.
- Experience with Azure Data Factory and Azure Data Lake Storage Gen2.
- Strong knowledge of the Microsoft Azure data ecosystem.
- Strong understanding of data governance, metadata management, data security, and access control.
- Mandatory experience within Investment Banking, Securities, or Capital Markets.
- Experience using Git and implementing CI/CD practices.
- Experience working in Agile/Scrum delivery environments.
- Excellent analytical, communication, and problem-solving skills.
- Must be willing to work onsite in Pittsburgh, PA.
- Must be willing and eligible to convert to full-time employment after six months.
- Databricks Certified Data Engineer Associate or Professional certification.
- Microsoft Certified Azure Data Engineer Associate, DP-203.
- Experience with Infrastructure as Code using Terraform or Bicep.
- Experience implementing DevOps practices for Azure Databricks environments.
- Familiarity with Microsoft Purview or Azure Purview.
#M1
#DI-CB2
#L1 - KB1
Ref: #404-IT Pittsburgh