Sr. Data Engineer (AWS, Databricks, SQL)

Very strong SQL, Databricks, Python, AWS Cloud, Medallion Architect

Sr. Data Engineer (AWS, Databricks, SQL)

Location: Bethlehem, PA / Holmdel NJ / New York, NY (Hybrid/ once in a week)

Nature of Contract: Contract opportunity

JOB DESCRIPTION:

We are seeking a hands-on-keyboard Data Engineer to build core data assets for a new third-party billing, commission, and claims application. You will design and execute data pipelines to move data into a Databricks lakehouse environment, creating reusable, reporting-ready data structures to support analytics and reporting.

Core Mission:

Move and transform data from source systems into a Databricks-based client Data Lake using the Medallion Architecture (Bronze -> Silver layers). Build standardized, domain-aligned tables-not one-off datasets-to serve 40-50 planned reports and future analytics.

Key Responsibilities:

• Write complex, optimized SQL to interrogate Oracle databases and explore raw data in Databricks.
• Develop and deploy production Python/PySpark scripts to build and extend data transformation pipelines.
• Analyze data attributes for planned reports; identify gaps and extend schemas accordingly.
• Build reusable, generalized data assets following architectural standards for a warehouse-like layer.
• Work within AWS cloud infrastructure (Databricks on AWS) and follow existing frameworks and patterns.
• Collaborate with data architects and participate in peer code reviews.

Technical Requirements (Must-Haves):

• Expert-level SQL proficiency This is the most critical skill.
• Hands-on development experience with Python and PySpark for building data pipelines.
• Practical experience with Databricks and the Medallion Architecture (Bronze/Silver/Gold layers).
• AWS cloud platform experience.
• Strong data analysis skills: ability to explore data and perform attribute-level analysis.
• Experience understanding object-oriented data models and working with transactional databases (e.g., Oracle).