What are the responsibilities and job description for the Lead Data Engineer position at Expert Technology Services?
Job Summary (Lead Data Engineer)
- Lead the design, development, and deployment of large-scale data engineering solutions.
- Provide technical leadership and guidance to data engineering teams.
- Architect, build, and maintain enterprise-level ETL/ELT pipelines using Python and Google Cloud Platform (GCP) tools.
- Ensure high performance and scalability of data processing workflows with BigQuery, Dataflow, Dataproc/PySpark, Cloud Composer (Airflow), Pub/Sub, and Cloud Storage.
- Develop and optimize complex SQL queries and data models to support business analytics and data science.
- Collaborate with stakeholders to translate business requirements into robust data solutions.
- Implement CI/CD, DevOps practices, and automated deployment frameworks for data engineering projects.
- Drive initiatives independently from concept through to production, ensuring best practices in coding, testing, and deployment.
- Support data governance, data quality, and compliance requirements as needed.
- Stay current with emerging technologies, especially in cloud data engineering and AI/ML enablement.
- (Preferred) Leverage experience with Snowflake, Databricks, lakehouse architectures, and feature engineering pipelines.
- (Preferred) Contribute to Agile processes within large enterprise environments.
- Lead the design, development, and deployment of large-scale data engineering solutions.
- Provide technical leadership and guidance to data engineering teams.
- Architect, build, and maintain enterprise-level ETL/ELT pipelines using Python and Google Cloud Platform (GCP) tools.
- Ensure high performance and scalability of data processing workflows with BigQuery, Dataflow, Dataproc/PySpark, Cloud Composer (Airflow), Pub/Sub, and Cloud Storage.
- Develop and optimize complex SQL queries and data models to support business analytics and data science.
- Collaborate with stakeholders to translate business requirements into robust data solutions.
- Implement CI/CD, DevOps practices, and automated deployment frameworks for data engineering projects.
- Drive initiatives independently from concept through to production, ensuring best practices in coding, testing, and deployment.
- Support data governance, data quality, and compliance requirements as needed.
- Stay current with emerging technologies, especially in cloud data engineering and AI/ML enablement.
- (Preferred) Leverage experience with Snowflake, Databricks, lakehouse architectures, and feature engineering pipelines.
- (Preferred) Contribute to Agile processes within large enterprise environments.