What are the responsibilities and job description for the Lead Databricks Engineer position at SDH Systems?
Lead Data Engineer
Location- San Jose, CA- Onsite
Contract role
Requirements
We are seeking a Lead Data Engineer with expertise in Databricks and Data Warehousing to drive data architecture, pipeline development, and optimization efforts. The ideal candidate will play a key role in designing scalable solutions, implementing best practices, and leading data initiatives within a dynamic and collaborative environment.
Key Responsibilities:
- Design, build, and optimize scalable data pipelines using Databricks, Apache Spark, and Azure technologies.
- Architect data warehousing solutions, ensuring seamless integration with cloud platforms and structured/unstructured data sources.
- Collaborate with business stakeholders to understand data needs and develop high-performance analytical solutions.
- Implement ETL/ELT processes leveraging cloud-based technologies such as Azure Data Factory, Snowflake, and Delta Lake.
- Ensure data quality, governance, and security compliance while managing large datasets efficiently.
- Drive performance tuning and optimization for data pipelines, ensuring efficiency across systems.
- Work closely with cross-functional teams to support machine learning and advanced analytics initiatives.
- Provide technical leadership and mentorship to junior data engineers, fostering a culture of innovation and continuous improvement.
- Stay updated on emerging data technologies and recommend strategies to enhance existing architectures.
Qualifications:
- 8 years of experience in data engineering, big data processing, and cloud-based solutions.
- Strong expertise in Databricks, Spark (PySpark/SQL), and Delta Lake architecture.
- Proven experience in designing and managing data warehouses using Snowflake, Azure Synapse, or equivalent technologies.
- Deep understanding of data modeling, SQL, and performance optimization.
- Hands-on experience with Azure Data Factory, Event Hubs, and cloud-based ETL processes.
- Solid knowledge of real-time streaming technologies (Kafka, Azure Stream Analytics, or similar).
- Familiarity with ML/AI data pipelines and feature engineering best practices.
- Strong communication and collaboration skills, with experience working in fast-paced, enterprise environments.
Salary : $60 - $65