Senior Data Engineer

Sr Data Engineer

Location: Charlotte or Raleigh

Core Technical & Functional Requirements

• 10 years of experience in data engineering, data platform development, or related technical fields.
• 3–5 years of hands‑on experience building solutions with Databricks and Apache Spark.
• Strong command of SQL, including complex transformations, optimization, and analytical querying.
• Proven experience designing and maintaining ETL/ELT pipelines, data models, and performance‑tuned workflows.
• Proficiency in Python or Scala for data engineering and automation tasks.
• Ability to troubleshoot complex data issues and communicate clearly with both technical and non‑technical stakeholders.

Role Overview

We are looking for a highly skilled Databricks Data Engineer who can design, build, and optimize data pipelines and analytics workflows in a cloud environment. This role requires strong engineering fundamentals, hands‑on development experience, and the ability to operate in a fast‑moving environment. Experience with insurance data or Power BI is helpful but not required.

Key Responsibilities

Data Pipeline & Workflow Development

• Build and maintain scalable ETL/ELT pipelines using Spark, PySpark, and Databricks.
• Develop efficient SQL transformations to support analytics, reporting, and downstream applications.
• Partner with product owners, architects, and analysts to design data models, Delta Lake structures, and end‑to‑end workflows.
• Implement orchestration, scheduling, and monitoring for data jobs and pipelines.
• Ensure data quality, reliability, and performance across all stages of the pipeline.
• Develop and maintain CI/CD processes for Databricks notebooks, jobs, and workflows.
• Work within cloud‑based data platforms (Azure preferred; AWS/GCP acceptable).

Collaboration & Delivery

• Work closely with business teams and data analysts to translate requirements into scalable engineering solutions.
• Support production environments, troubleshoot issues, and ensure high availability of data pipelines.
• Contribute to best practices around coding standards, documentation, and operational excellence.

Required Skills & Experience

• 10 years in data engineering or related technical roles.
• 3–5 years of hands‑on Databricks and Spark development.
• Strong SQL expertise and analytical problem‑solving skills.
• Experience with ETL/ELT design, data modeling, and performance tuning.
• Proficiency in Python or Scala for data engineering.
• Strong communication skills and the ability to work cross‑functionally.

Nice to Have

• Familiarity with insurance industry data structures or workflows.
• Exposure to Power BI or similar BI tools.