What are the responsibilities and job description for the Sr. Data Fabric Engineer 5 days on-site position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, SDH Systems, is seeking the following. Apply via Dice today!
Senior Data Fabric Engineer
Location: San Jose, CA | 5 days on-site
Contract role
Job Description:
Role Overview
We are seeking a highly skilled Senior Data Engineer with a deep specialization in Apache Spark and the Microsoft Fabric ecosystem. The ideal candidate isn''t just a notebook user; you are an expert in developing, packaging, and deploying Spark-based JAR files (Scala/Java) to handle complex, high-scale data processing requirements.
You will be responsible for architecting robust ETL/ELT pipelines, optimizing Spark performance, and leveraging the full suite of Microsoft Fabric tools (OneLake, Lakehouse, and Data Factory) to drive our data strategy forward.
Key Responsibilities
Core Essentials
Senior Data Fabric Engineer
Location: San Jose, CA | 5 days on-site
Contract role
Job Description:
Role Overview
We are seeking a highly skilled Senior Data Engineer with a deep specialization in Apache Spark and the Microsoft Fabric ecosystem. The ideal candidate isn''t just a notebook user; you are an expert in developing, packaging, and deploying Spark-based JAR files (Scala/Java) to handle complex, high-scale data processing requirements.
You will be responsible for architecting robust ETL/ELT pipelines, optimizing Spark performance, and leveraging the full suite of Microsoft Fabric tools (OneLake, Lakehouse, and Data Factory) to drive our data strategy forward.
Key Responsibilities
- Custom Spark Development: Design and develop high-performance data processing applications using Scala or Java, compiled into JARs for execution on Spark clusters.
- Fabric Implementation: Architect and maintain end-to-end data solutions within Microsoft Fabric, utilizing Synapse Data Engineering and OneLake.
- Performance Tuning: Optimize Spark jobs by managing partitions, caching strategies, and memory management to ensure efficient resource utilization.
- Pipeline Orchestration: Build and automate sophisticated data workflows using Fabric Data Factory and Airflow (or similar).
- DevOps & CI/CD: Manage the lifecycle of Spark JARs through automated CI/CD pipelines (Azure DevOps/GitHub Actions), ensuring seamless deployment to Fabric environments.
- Data Modeling: Implement Medallion Architecture (Bronze/Silver/Gold) and maintain Delta Lake tables for ACID compliance and time travel capabilities.
Core Essentials
- Spark Expertise: 5 years of experience with Apache Spark, specifically in writing and deploying compiled JAR files rather than solely relying on PySpark notebooks.
- Languages: Proficiency in Scala or Java (required) and Python/SQL (preferred).
- Platform: Hands-on experience with Microsoft Fabric or migrations from Azure Databricks/Synapse to Fabric.
- Storage: Deep understanding of Delta Lake format and Parquet optimization.