What are the responsibilities and job description for the Lead Data Engineer - Airflow, Trino position at Photon?
Technical Lead Data Engineering
Position Overview
We are seeking an experienced Technical Lead Data Engineering to lead a team of 5 6 data engineers and drive the design, development, and implementation of scalable, high-performance data engineering solutions.
The ideal candidate will possess strong hands-on expertise in Python, Apache Airflow, Snowflake, Trino, Hive, Apache Kafka, and SQL, along with experience in data pipeline development, workflow orchestration, and distributed data processing. This role requires a balance of technical leadership, hands-on engineering, performance optimization, and team mentoring.
Key Responsibilities
Lead, mentor, and guide a team of 5 6 data engineers, fostering technical excellence, collaboration, and continuous improvement.
Design, develop, and maintain scalable, reliable, and efficient data pipelines using Python and Apache Airflow.
Develop and manage Airflow DAGs for workflow orchestration, scheduling, monitoring, and dependency management. Experience with Apache Airflow 3 is preferred.
Design, implement, and optimize data processing solutions using Snowflake, Trino, and Apache Hive.
Work with Snowflake architecture, including data storage, compute, data loading, and query performance optimization.
Develop and optimize complex SQL queries across data platforms, ensuring efficient data retrieval and processing.
Leverage Trino for distributed SQL query execution and data access across multiple data sources.
Utilize Hive for large-scale data processing, querying, and management of data stored in distributed environments.
Build and maintain real-time streaming and event-driven data pipelines using Apache Kafka and Spark Streaming.
Develop Python-based data transformation, integration, and automation solutions.
Build and integrate REST/SOAP APIs to facilitate data exchange across multiple systems and platforms.
Drive technical design decisions, establish coding standards, and implement data engineering best practices.
Conduct code reviews, troubleshoot complex technical issues, and ensure data quality, reliability, and performance.
Collaborate with cross-functional teams, business stakeholders, and architects to translate requirements into effective technical solutions.
Follow Agile/Scrum methodologies and ensure timely delivery of high-quality engineering solutions.
Required Qualifications
7 8 years of hands-on experience in data engineering, data platform development, or related fields.
3 years of technical leadership experience preferred, including mentoring engineers and guiding technical delivery.
Strong programming skills in Python, including data processing, scripting, automation, and pipeline development.
Hands-on experience with Apache Airflow, including DAG development, workflow orchestration, scheduling, and monitoring. Airflow 3 experience is preferred.
Strong understanding of Snowflake architecture, data warehousing concepts, and query optimization.
Practical experience with Trino, including distributed SQL querying and data access across multiple data sources.
Experience with Apache Hive, including HiveQL, data querying, and large-scale data processing.
Advanced SQL skills, including complex queries, joins, analytical functions, and performance tuning.
Hands-on experience with Apache Kafka and event-driven data architectures.
Experience with Spark Streaming or similar real-time data processing technologies.
Knowledge of REST/SOAP API development, integration, and data exchange across systems.
Understanding of scalable data pipeline design, data integration, data transformation, and data quality practices.
Strong problem-solving, analytical, communication, and stakeholder management skills.
Experience working in Agile/Scrum environments.
Preferred Qualifications
Experience with Apache Airflow 3.
Experience designing and optimizing distributed data processing and query execution workflows.
Exposure to cloud-based data platforms and modern data engineering architectures.
Experience establishing engineering standards, improving pipeline performance, and mentoring technical teams.
What We Offer
Opportunity to lead impactful data engineering initiatives and modern data platform development.
A collaborative and innovative work environment.
Opportunities to drive technical strategy, mentor engineering talent, and implement scalable data solutions.