What are the responsibilities and job description for the Sr. Data Fabric Engineer 5 days on-site position at SDH Systems?
Senior Data Fabric Engineer
Location: San Jose, CA | 5 days on-site
Contract role
Job Description:
Role Overview
We are seeking a highly skilled Senior Data Engineer with a deep specialization in Apache Spark and the Microsoft Fabric ecosystem. The ideal candidate isn''t just a notebook user; you are an expert in developing, packaging, and deploying Spark-based JAR files (Scala/Java) to handle complex, high-scale data processing requirements.
You will be responsible for architecting robust ETL/ELT pipelines, optimizing Spark performance, and leveraging the full suite of Microsoft Fabric tools (OneLake, Lakehouse, and Data Factory) to drive our data strategy forward.
Key Responsibilities
- Custom Spark Development: Design and develop high-performance data processing applications using Scala or Java, compiled into JARs for execution on Spark clusters.
- Fabric Implementation: Architect and maintain end-to-end data solutions within Microsoft Fabric, utilizing Synapse Data Engineering and OneLake.
- Performance Tuning: Optimize Spark jobs by managing partitions, caching strategies, and memory management to ensure efficient resource utilization.
- Pipeline Orchestration: Build and automate sophisticated data workflows using Fabric Data Factory and Airflow (or similar).
- DevOps & CI/CD: Manage the lifecycle of Spark JARs through automated CI/CD pipelines (Azure DevOps/GitHub Actions), ensuring seamless deployment to Fabric environments.
- Data Modeling: Implement Medallion Architecture (Bronze/Silver/Gold) and maintain Delta Lake tables for ACID compliance and time travel capabilities.
Technical Requirements
Core Essentials
- Spark Expertise: 5 years of experience with Apache Spark, specifically in writing and deploying compiled JAR files rather than solely relying on PySpark notebooks.
- Languages: Proficiency in Scala or Java (required) and Python/SQL (preferred).
- Platform: Hands-on experience with Microsoft Fabric or migrations from Azure Databricks/Synapse to Fabric.
- Storage: Deep understanding of Delta Lake format and Parquet optimization.
Salary : $60 - $70