Job Title: Lead Data Engineer

Location: Scottsdale, AZ (100% onsite)

Hire Type: Contract/FTE/CTH

VISA: No OPT, CPT, H1B

Must-have skill set:

• Spark, S3, Glue, AWS Redshift, Python, and StreamSets experience

Description:

We are seeking a highly skilled Lead Data Engineer to join our team in Scottsdale, AZ. The ideal candidate will have extensive experience in enterprise data architecture and management, with proficiency in Spark, S3, Glue, AWS Redshift, Python, and StreamSets.

Responsibilities:

• Lead and architect enterprise-wide initiatives including system integration, data migration, transformation, data warehouse build, data mart build, and data lakes implementation/support.
• Conceptualize, design, and implement streaming data pipelines using Spark, Structured Streaming, Delta Lake, and Databricks.
• Develop and maintain data models including conceptual, logical, and physical data modeling, with expertise in relational and dimensional data modeling.
• Troubleshoot and debug complex technical issues related to data pipelines and infrastructure.
• Collaborate with cross-functional teams to ensure the successful deployment and optimization of big data applications in a high-performance enterprise environment.
• Utilize advanced SQL skills for data manipulation, performance optimization, and RDBMS schema design.
• Implement and maintain ETL/ELT processes using tools such as Data Migration Service.
• Work with AWS environment, Lambda, S3, and other cloud services for data processing and storage.
• Familiarity with data quality/validation frameworks such as Great Expectations is a plus.
• Experience with Gitlabs, CloudWatch, CI/CD pipelines, and Lambda configuration and optimization.
• Communicate effectively with technology partners, stakeholders, team members, and senior management to articulate technology solutions and constraints.

Qualifications:

• Bachelor's degree in computer science, Information Technology, or related field.
• 6-8 years of IT experience focusing on enterprise data architecture and management.
• Proficiency in Spark, S3, Glue, AWS Redshift, Python, and StreamSets.
• Experience with Databricks, Structured Streaming, Delta Lake, and Delta Live Tables.
• Advanced understanding of streaming data pipelines and ETL/ELT processes.
• Strong SQL skills with experience in RDBMS schema design and optimization.
• Experience working in a team-based environment and ability to thrive in a fast-paced, dynamic organization.

If you possess the required skills and experience and are eager to join a dynamic team at Persistent Systems, we encourage you to apply for this exciting opportunity.