What are the responsibilities and job description for the PySpark Data Engineer position at Covetus?
Title: PySpark Data Engineer
Location: Irving, TX (Onsite)
Employment Type: Full Time
Job Description:
As a Data Engineer, you will be responsible for designing, developing, and maintaining data solutions for data generation, collection, and processing in Big Data environment using predominantly PySpark/Python. Your typical day will involve creating data pipelines, ensuring data quality, and implementing ETL processes to migrate and deploy data across systems using PySpark.
Roles & Responsibilities:
• Design, develop, and maintain robust, scalable high-performance Data Pipelines using PySpark.
• Create data pipelines, ensuring data quality, and implement ETL processes to migrate and deploy data across systems.
• Migrate Ab Initio ETL Applications into Pyspark based data pipelines
• Migrate On Prem workloads into Cloud(AWS, Databricks, Snowflake) based on use cases.
• Collaborate with cross-functional teams to identify and resolve data-related issues.
• Stay updated with the latest advancements in data engineering and integrate innovative approaches for sustained competitive advantage.
Qualifications:
• 8 years of professional experience in Hadoop, PySpark/Python development.
• Proven expertise in PySpark and experience in handling huge volume of data
• 3 working experience in AWS, Databricks/Snowflake, Airflow
• Familiarity with CI/CD pipelines and version control systems (e.g., Git).
• Strong debugging and problem-solving skills.
• Excellent communication and collaboration skills.
Good to Have Skills:
• AWS EKS Experience, Dockers and Containers.
Salary : $125,000 - $140,000