Lead Data Engineer

Job Title: Lead Data Engineer

Location: Irvine/Los Angeles (3 days hybrid)

Exerience: 10 years overall, 3-4 years in specified technologies

Rate : depends on evaluation, pls share your expected /hr rate

Job Description:

We are seeking a highly skilled and experienced Lead Data Engineer to join our dynamic team. The ideal candidate will have a strong background in data engineering, with extensive experience in Python, PySpark, AWS services, PostgreSQL, Apache Airflow and Docker.

This role requires a professional with a proven track record of designing, implementing, and maintaining robust data pipelines and architectures.

Key Responsibilities:

• Design and Develop Data Pipelines: Create and maintain scalable data pipelines using Python and PySpark to process large volumes of data efficiently.
• Cloud Integration: Utilize AWS services (such as S3, CloudWatch, ECS, ECR, Lambda) to build and manage cloud-based data solutions.
• Database Management: Design, implement, and optimize PostgreSQL databases to ensure high performance and reliability.
• Workflow Orchestration: Use Apache Airflow to schedule and monitor complex data workflows.
• Containerization: Implement and manage Docker containers to ensure consistent and reproducible environments for data processing tasks.
• Data Quality and Governance: Ensure data quality, integrity, and security across all data pipelinesand storage solutions.
• Collaboration: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver solutions that meet business needs.
• Mentorship: Provide guidance and mentorship to junior data engineers and contribute to the continuous improvement of the team s skills and processes.

Qualifications:

Education: Bachelor s or Master s degree in Computer Science, Engineering, or a related field.

Experience: 10 years of overall experience in data engineering or related fields.

Technical Skills:

• Proficiency in Python and PySpark.
• Extensive experience with AWS services (S3, CloudWatch, ECS, ECR, Secrets Manager, Cloud9 IDE).
• Strong knowledge of PostgreSQL and database optimization techniques.
• Hands-on experience with Apache Airflow for workflow orchestration.
• Proficiency in Docker for containerization.

Soft Skills:

• Excellent problem-solving and analytical skills.
• Strong communication and collaboration abilities.
• Ability to work in a fast-paced, dynamic environment.
• Preferred Qualifications:
• Familiarity with CI/CD pipelines and DevOps practices.
• Knowledge of data warehousing concepts and ETL processes.