Role & Responsibilities Overview:

• Develop and optimize ETL pipelines from various data sources using Databricks on cloud (AWS, Azure, etc.)
• Experienced in implementing standardized pipelines with automated testing, Airflow scheduling, Azure DevOps for CI/CD, Terraform for infrastructure as code, and Splunk for monitoring
• Continuously improve systems through performance enhancements and cost reductions in compute and storage
• Data Processing and API Integration: Utilize Spark Structured Streaming for real-time data processing and integrate data outputs with REST APIs
• Lead Data Engineering Projects to manage and implement data-driven communication systems
• Experienced with Scrum and Agile Methodologies to coordinate global delivery teams, run scrum ceremonies, manage backlog items, and handle escalations
• Integrate data across different systems and platforms
• Strong verbal and written communication skills to manage client discussions

Candidate Profile:

• 8 years' experience in developing and implementing ETL pipelines from various data sources using Databricks on cloud
• Based out of US
• Some experience in insurance domain/ data is must
• Programming Languages SQL, Python
• Technologies - IaaS (AWS or Azure or GCP), Databricks platform, Delta Lake storage, Spark (PySpark, Spark SQL).
• Good to have - Airflow, Splunk, Kubernetes, Power BI, Git, Azure DevOps
• Project Management using Agile, Scrum
• B.S. Degree in a data-centric field (Mathematics, Economics, Computer Science, Engineering or other science field), Information Systems, Information Processing or engineering.
• Excellent communication & leadership skills, with the ability to lead and motivate team members
• Ability to work independently with some level of ambiguity and juggle multiple demands

Job Type: Full-time

Pay: $130,000.00 - $185,000.00 per year

Benefits:

• 401(k)
• 401(k) matching
• Dental insurance
• Health insurance
• Paid time off
• Vision insurance

Compensation package:

• Bonus opportunities
• Yearly pay

Experience level:

• 10 years
• 11 years
• 8 years
• 9 years

Schedule:

• 8 hour shift
• Monday to Friday

Ability to Relocate:

• Jersey City, NJ 07304: Relocate before starting work (Required)

Work Location: Hybrid remote in Jersey City, NJ 07304