Databricks Platform Lead / Technical Delivery Lead

Job Title: Databricks Platform Engineering Lead / Technical Delivery Lead

Work Location: Warren, NJ (Hybrid)

Employment Type: Fulltime/Permanent

Need Independent Candidates Only | Local or Nearby

Minimum Years of Experience: 10 Years

Project Overview

We are seeking an experienced Databricks Platform Engineering Led to support an enterprise-wide pricing application within a global reinsurance organization. The application leverages Databricks as its data and analytics platform and integrates with a SQL Server-based pricing application.

The resource will lead the engineering, governance, and ongoing enhancement of the Databricks platform supporting the pricing application. The role requires a strong combination of hands-on platform engineering, Databricks administration, data engineering, and technical delivery leadership.

The ideal candidate must be capable of managing the Databricks platform foundation while providing technical oversight to data-engineering teams building ETL/ELT pipelines, transformations, and pricing data products.

Technical and Delivery Expertise

• 10 years of overall technology experience.
• 5 years of hands-on experience with Databricks, cloud data platforms, or platform engineering.
• 3 years of experience in a Platform Lead, Databricks Lead, Cloud Platform Engineer, Data Platform Lead, or similar role.
• Strong hands-on experience with:
• Azure Databricks
• Databricks account and workspace administration
• Unity Catalog
• Apache Spark and PySpark
• Python and SQL
• Delta Lake and Lakehouse architecture
• Azure Data Lake Storage
• Azure Data Factory or comparable integration tools
• Databricks Workflows and job orchestration
• Infrastructure as code using Terraform
• CI/CD and automated deployment practices
• Strong understanding of:
• Databricks workspace provisioning and administration
• Development, test, and production environment management
• Compute provisioning and cluster configuration
• Cluster policies, pools, runtimes, and serverless compute
• Platform security, identity, and access management
• Service principals, managed identities, and secrets management
• Azure networking, private endpoints, and secure connectivity
• Unity Catalog architecture, permissions, lineage, and governance
• Platform monitoring, logging, auditability, and observability
• Databricks performance, capacity, and cost optimization
• Production support and platform-incident management
• Good understanding of data-engineering practices, including:
• Batch and streaming data ingestion
• ETL/ELT pipeline development
• Medallion architecture
• Data transformations and enrichment
• Data quality, validation, and reconciliation
• Schema management and data lineage
• Spark workload and query optimization
• Experience leading globally distributed platform, data-engineering, application, QA, DevOps, and production-support teams.
• Ability to participate in architecture reviews, solution-design discussions, code reviews, security assessments, and production-readiness evaluations.
• Strong understanding of Agile delivery, release management, risk management, dependency management, and stakeholder governance.

Preferred Qualifications

• Prior experience implementing a Databricks-enabled pricing, underwriting, actuarial, insurance analytics, or financial-services application.
• Experience working with global insurance or reinsurance carriers.
• Understanding of reinsurance pricing, underwriting, treaty, and facultative business processes.
• Experience integrating Databricks with:
• Pricing engines
• SQL Server applications
• Policy, claims, and accounting systems
• Data warehouses and reporting platforms
• Internal and third-party data sources
• Experience establishing reusable Databricks platform standards, engineering frameworks, and reference architectures.
• Experience implementing a hybrid solution of Databricks and SQL
• Experience defining the division of responsibility among platform engineering, data engineering, application development, DevOps, data science, and production-support teams.
• Familiarity with MLflow, model lifecycle management, model serving, or actuarial-model integration.
• Experience implementing data governance, metadata management, lineage, role-based access, audit controls, and regulatory controls.
• Familiarity with Azure DevOps, Terraform, Power BI, Microsoft Purview, Azure Monitor, or comparable tools.
• Databricks and Microsoft Azure certifications are preferred.
• Experience supporting business-critical production platforms with defined availability, recovery, security, and performance requirements.

The Resource Will Be Expected To:

• Own the engineering and operational management of the Databricks platform supporting the enterprise pricing application.
• Lead end-to-end technical delivery of platform enhancements, environment changes, integrations, releases, and production improvements.
• Provision and administer Databricks accounts, workspaces, compute resources, clusters, policies, and supporting Azure services.
• Define and enforce platform standards covering architecture, security, governance, compute, deployment, monitoring, and cost management.
• Design and maintain appropriate separation across development, test, UAT, and production environments.
• Implement infrastructure as code and automated deployment practices for Databricks and supporting cloud resources.
• Establish secure connectivity between Databricks, SQL Server, cloud storage, the pricing engine, the underwriting workbench, and other enterprise systems.
• Configure and maintain identity, access, security, secrets, networking, and governance controls.
• Define Unity Catalog structures, access models, data ownership, lineage, and auditability standards.
• Establish cluster policies, runtime standards, compute controls, and workload-management practices.
• Monitor and optimize platform availability, reliability, performance, capacity, and cost.
• Troubleshoot workspace, authentication, networking, compute, storage, access, and platform-performance issues.
• Lead root-cause analysis and corrective actions for platform incidents and recurring operational problems.
• Provide technical oversight to data engineers building ingestion pipelines, transformations, curated datasets, and pricing data products.
• Review pipeline designs, Spark workloads, data-processing patterns, deployment approaches, and production-readiness plans.
• Ensure that data-engineering solutions comply with platform, security, governance, quality, and performance standards.
• Coordinate platform engineering, data engineering, application development, QA, DevOps, infrastructure, security, and production-support teams.
• Independently identify technical and delivery impacts across Databricks, the SQL Server pricing engine, the underwriting workbench, and integrated systems.
• Support successful SIT, UAT, regression testing, release validation, deployment, cutover, and post-production stabilization.
• Communicate technical risks, delivery dependencies, architectural decisions, and recovery plans to business and technology stakeholders.
• Serve as the primary technical point of contact for the Databricks platform and its integration with the enterprise pricing application.
• Serve as a subject matter expert on Databricks platform engineering, administration, governance, security, reliability, and operational best practices.