What are the responsibilities and job description for the Senior Data Engineer position at Wise Skulls?
Hiring: Lead Data Engineer - Databricks
Important
Visa: USC and GC only
Location: Manhattan, New York (Onsite)
JD:
Role Overview
- Client is seeking a Lead Data Engineer with deep hands-on expertise in Databricks, SQL, Python, and
- PySpark to design, develop, and optimize scalable data engineering solutions.
- This role offers an opportunity to work on enterprise-scale data platforms supporting analytics, digital
- products, and emerging AI use cases. The candidate will drive Databricks engineering best practices,
- solve challenging technical problems, and provide technical leadership across multiple initiatives.
Key Responsibilities
- Design and develop scalable data pipelines on Databricks, following Medallion Architecture.
- Build and optimize data ingestion, transformation, and orchestration workflows for batch and
- incremental processing.
- Optimize data processing workloads, SQL queries, and compute configurations for performance
- and cost efficiency.
- Ensure data quality, observability, lineage, and reliability across production pipelines, including
- troubleshooting complex issues.
- Provide hands-on technical leadership through solution design, code reviews, reusable
- engineering patterns, and mentoring other engineers.
Required Qualifications
- 6 years of Data Engineering experience, with substantial hands-on experience delivering
- production solutions on Databricks.
- Strong proficiency in SQL, Python, and PySpark, including distributed data processing and
- performance optimization.
- Strong understanding of Databricks Lakehouse architecture, Delta Lake, Unity Catalog, and
- workflow orchestration.
- Familiarity with Git, CI/CD, and production deployment practices.
- Strong troubleshooting, problem-solving, and communication skills.
Preferred Qualifications
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience working with AWS cloud services, particularly S3 and cloud-based data integrations.
- Experience with advanced ingestion patterns such as Auto Loader, Structured Streaming, or
- Change Data Capture (CDC).
- Familiarity with Databricks Asset Bundles or Apache Airflow.
- Experience with large-scale media datasets: streaming, clickstream, or advertising data.
Interested? Apply now or DM us to explore this opportunity! You can share resumes at taukir@wiseskulls.com