Lead Data Engineer position is a key engineering role helping us to design, build, and support our future state data platform. In this role you will be expected to spearhead the design, development, and implementation of our next-generation data architecture, while leading and coaching junior engineers. This role also requires a combination of technical expertise & strategic thinking as you will be collaborating closely with cross-functional teams, to bridge the gap between business requirements and technical solutions.

Qualification

• 9 years of experience as a Data Engineer, with at least 3 years in a lead role.
• Strong proficiency Python, Scala, SQL, or HQL for data manipulation, analysis, and automation of tasks
• Proven experience designing and implementing data architectures on AWS, Azure, or GCP.
• Experience and Comfort working with large scale data sets. Petabyte scale, with daily ingress and egress numbers in the 10’s of Terabytes
• Expertise in managed cloud services for data storage, ingestion, and processing. Will need to articulate the services used for various stages of the data lifecycle.
• Experience with notebook environments and big data engineering tools such as: Databricks, Sage Maker, Google Colab, Jupyter, etc
• Hands on experience with streaming tools such as: Kafka, Flume, Spark Streaming, or Flink
• Expectation is that you support the code and products yourself and your team create

What You’ll Do:

• Engineer an innovative, durable, robust, and economically viable data platform enabling business velocity, delightful user experiences, and timely/meaningful analytics
• Design and develop data pipelines to manage how data flows between disparate systems.
• Develop data quality metrics and perform QC tests (system and visual) to verify data integrity.
• Interface with architects, product managers/SMEs and product analysts to understand data needs and support the implementation of the business rules into transformation.
• Document the data blending process along with the specifications and workflow/data lineage.
• Perform continuous integration to ensure that every step of the pipeline is testable and automated
• Perform detailed design (the Physical Data Model and transformations), based on understanding of the Logical Data Model (the business requirements)
• Create design documents for data integration or data reporting projects
• Analyze data integration problems, provide solutions and recommend corrective actions