Job Description: Sr. Data Engineer

Introduction:

As a Sr. Data Engineer, you will be an integral part of our data team, responsible for designing, implementing, and maintaining data pipelines and infrastructure. You will work closely with data scientists, analysts, and other stakeholders to ensure data quality, accessibility, and reliability.

Responsibilities:

• Design and develop scalable data pipelines using tools such as Hive, Airflow, NiFi, and Scoop.
• Work with Hadoop and Cloudera to manage and optimize big data infrastructure.
• Collaborate with cross-functional teams to understand data requirements and implement solutions.
• Ensure data quality and integrity through data validation and testing processes.
• Troubleshoot and resolve data pipeline issues in a timely manner.
• Stay current with industry trends and best practices in data engineering.

Requirements:

Required Skills:

• Proficiency in Hive for data warehouse management.
• Experience with Airflow for workflow orchestration.
• Knowledge of NiFi for data ingestion and processing.
• Familiarity with Scoop for data transfer between Hadoop and relational databases.
• Strong understanding of Hadoop and Cloudera for big data processing.

Preferred Skills:

• Proficiency in programming languages such as Python and Scala for data manipulation and analysis.
• Experience with data modeling and ETL processes.
• Ability to work in a fast-paced environment and manage multiple projects simultaneously.
• Excellent communication and collaboration skills.