What are the responsibilities and job description for the Lead Spark Scala Engineer position at Purple Drive Technologies LLC?
Job Title: Lead Spark Scala Engineer
Role Summary
We are seeking a Lead Spark Scala Engineer with strong expertise in Apache Spark and Scala to design, build, and optimize largescale data platforms. The ideal candidate must have prior Apple experience and proven ability to lead technical execution, manage distributed systems, and drive strategic initiatives across complex data environments.
Key Responsibilities
-
Design and develop Spark applications using Scala DataFrame APIs.
-
Build and manage applications for largescale data platforms including storage, ingestion, governance, and performance optimization.
-
Troubleshoot and resolve complex issues in distributed systems.
-
Collaborate with crossfunctional teams to deliver on status reporting, roadmap reviews, quarterly engineering reviews, and newsletters.
-
Drive technical execution and ensure alignment with strategic objectives.
-
Analyze and interpret data transformations, ensuring accuracy and consistency in analytics and metrics.
-
Provide leadership in project management, communication, and stakeholder engagement.
Essential Skills
-
Strong experience in Apache Spark with Scala.
-
Handson expertise with Scala DataFrame APIs.
-
Knowledge of AWS/S3, Iceberg, Git.
-
Strong SQL and data modeling skills.
-
Experience building/managing applications for largescale data platforms.
-
Skilled at troubleshooting distributed systems.
-
Strong understanding of technical concepts and ability to think strategically.
-
Prior Apple experience (mandatory).
-
Familiarity with using a Mac laptop.
-
Excellent communication and project management skills.
Preferred Skills
-
Experience with data analytics and interpretation beyond data movement.
-
Ability to identify how data transformations impact downstream analytics and metrics.
-
Strong strategic thinking and ability to handle fastpaced, ambiguous projects with multiple stakeholders.
Experience Required
-
6-8 years of experience in Spark/Scala engineering.
-
Proven track record in largescale data platform development.
-
Prior Apple project experience is mandatory.