What are the responsibilities and job description for the Data Platform Engineer position at BENNETT MOTOR EXPRESS?
Data Platform Engineer
|
Department: Data & Analytics |
Location: Hybrid |
Employment Type: Full-Time |
Position Overview
We are seeking an experienced Data Engineer to join our growing team and play a central role in shaping our data strategy. This individual will be responsible for building, optimizing, and governing our Databricks-based data platform hosted on AWS connecting disparate data sources, enabling reliable data pipelines, and ensuring that high-quality data is available to support business intelligence, analytics, and machine learning initiatives.
Key Responsibilities
Data Strategy & Platform Governance
- Partner with leadership to define and execute a comprehensive data strategy aligned with business goals.
- Establish standards and best practices for data ingestion, transformation, storage, and access across the organization.
- Drive adoption of data governance frameworks including data cataloging, lineage tracking, and quality monitoring.
- Champion a culture of data-driven decision-making by enabling self-service analytics capabilities.
Databricks Platform Administration
- Architect, configure, and maintain the Databricks environment hosted on AWS (EMR, S3, IAM, VPC, etc.).
- Manage Databricks workspaces, clusters, Unity Catalog, and access controls.
- Optimize compute and storage costs through cluster sizing, auto-scaling, and lifecycle management.
- Implement and maintain CI/CD pipelines for data workflows using tools
Data Integration & Pipeline Development
- Design and build scalable ETL/ELT pipelines to ingest data from a variety of sources including SaaS applications, databases, APIs, and streaming platforms.
- Leverage Delta Lake, Delta Live Tables, and Databricks Workflows to build reliable, incremental data processing solutions.
- Integrate cloud-native AWS services (Glue, Kinesis, EventBridge, Lambda) with the Databricks ecosystem.
- Ensure data pipeline reliability through monitoring, alerting, and automated recovery mechanisms.
Collaboration & Data Enablement
- Partner with data analysts, data scientists, and business stakeholders to understand data requirements and deliver fit-for-purpose datasets.
- Develop and maintain data documentation, including data dictionaries, runbooks, and architecture diagrams.
- Mentor junior team members on best practices in data engineering and cloud-native development.
Required Qualifications
Technical Skills
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Familiarity with data transformation tooling such as dbt, Apache Spark, or similar frameworks. |
|
|
|
Experience & Education
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
We are not able to consider candidates who will require sponsorship at any point during employment.
EEO/Vets/Women/Minorities/Disabled
www.bennettig.com