What are the responsibilities and job description for the Lead Data Engineer to San Francisco, CA (Hybrid) - Local to CA Only position at BayOne Solutions?
ROLE: DATA ENGINEER LEAD (EDW to DATABRICKS MIGRATION)
LOCATION: SAN FRANCISCO, CA (HYBRID)
Final Interview will be F2F in client office.
JOB OVERVIEW:
- Seeking a highly experienced Senior Data Engineer to support a large-scale enterprise data warehouse modernization initiative. The role focuses on migrating a 20-year-old SQL-based EDW environment to Databricks, involving deep analysis of legacy ETL pipelines, data models, and reporting dependencies. This is not a lift-and-shift migration the work requires full re-engineering of ETL pipelines, redesigning data models, and modernizing reporting integrations.
- The ideal candidate is both architecturally strong and hands-on, with extensive experience in SQL, Datastage, Python, and Databricks, and has previously worked on complex data platform migration projects.
JOB DESCRIPTION:
- We are looking at 8-10 years of software development and deployment experience with at least 7 years of hands-on experience with SQL, Databricks, ADF, DataStage (or other ETL tool).
- Write SQL for processing raw data, Kafka ingestions, ADF pipelines, data validation, and QA.
- Knowledge working with APIs to collect or ingest data.
- Experience with advanced SQL and Python.
- Strong Database knowledge, SQL & No-SQL preferred.