What are the responsibilities and job description for the Data Engineer position at ATC?
Job Summary
We are seeking a Data Engineer with 5 years of experience in designing, developing, and optimizing scalable data pipelines and cloud-based data platforms. The ideal candidate will have strong expertise in Python, SQL, Databricks, Apache Spark, and ETL/ELT development, along with hands-on experience in at least one cloud platform (AWS, Azure, or Google Cloud Platform).
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT data pipelines.
- Build and optimize data ingestion, transformation, and integration workflows using Databricks and Apache Spark.
- Develop and maintain data lakes and cloud-based data warehouses.
- Create scalable batch and real-time data processing solutions.
- Optimize data pipelines for performance, reliability, and scalability.
- Develop reusable data engineering frameworks and automation solutions.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver high-quality data solutions.
- Implement data quality, governance, monitoring, and security best practices.
- Troubleshoot production issues and continuously improve data platform performance.
- Follow DevOps and CI/CD best practices for data engineering projects.
Required Qualifications
- Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, or a related field.
- 5 years of experience as a Data Engineer.
- Strong programming skills in Python.
- Advanced SQL proficiency.
- Hands-on experience with Databricks.
- Strong experience with Apache Spark (PySpark preferred).
- Experience building and maintaining ETL/ELT pipelines.
- Experience with Apache Airflow or similar workflow orchestration tools.
- Experience with data warehousing technologies such as Snowflake, Amazon Redshift, Google BigQuery, or Azure Synapse Analytics.
- Hands-on experience with at least one cloud platform (AWS, Azure, or Google Cloud Platform).
- Experience with Git, Docker, and CI/CD pipelines.
- Knowledge of relational and NoSQL databases.
Preferred Qualifications
- Experience with Kafka or other streaming platforms.
- Experience with Delta Lake.
- Familiarity with Apache Iceberg or Apache Hudi.
- Experience with Infrastructure as Code (Terraform or CloudFormation).
- Knowledge of data governance and data quality frameworks.
- Exposure to DevOps and MLOps practices.