DataOps Engineer

We are looking for a midlevel engineer to build and operate a data platform that uses Apache Iceberg as the lakehouse table format and Dockerbased microservices (Spark, Flink, Presto, etc.). you will own the endtoend delivery pipeline, monitoring, security, and incident response, ensuring the platform runs reliably at scale.

Key Responsibilities

• Iceberg operations: support tables, manage schema changes, partitions, snapshot retention, and keep the catalog (Hive Metastore, AWSGlue, Nessie, ...) synchronized.
• Docker image creation & testing: write multistage Dockerfiles for Spark/Flink/Presto, run local test environments with DockerCompose, and conduct vulnerability scans (Trivy, Snyk, ...).
• Data pipeline development: build ETL/ELT jobs that ingest raw data and write to Iceberg tables; add simple streaming components using Kafka, Pulsar, or Kinesis when needed.
• CI/CD automation: configure pipelines (GitHubActions, GitLabCI, AzureDevOps, ...) to lint Dockerfiles, scan images, version Iceberg metadata, and deploy pipelines without downtime.
• Automation with Ansible/Python: script cluster provisioning, catalog configuration, vacuum/compaction, and other routine housekeeping tasks.
• Observability: instrument services with OpenTelemetry, Prometheus, Grafana, and Loki; create dashboards showing pipeline latency, resource usage, table health, and error rates; set up basic alerts.
• SLA monitoring: measure data freshness, job success rates, and query response times against agreedupon targets and report deviations.
• Incident response: join the oncall rotation, perform firstline diagnosis and resolution of pipeline failures, Iceberg metadata issues, or container crashes; write concise rootcause analyses and suggest improvements.
• Security & compliance support: help enforce image signing, mTLS, IAM roles, and bucket policies; collaborate with the security team to meet GDPR, HIPAA, or ISO27001 requirements.
• Knowledge sharing: keep internal documentation up to date and run short tech demos or brownbag sessions on Iceberg, Docker best practices, and automation techniques.

Minimum Requirements

• Bachelor's degree in Computer Science, IT, Data Engineering, or a related field (Master's a plus).
• ~5years of handson experience building and operating largescale data platforms (lakehouse, datawarehouse, or bigdata ecosystems).
• Proven production experience with Apache Iceberg (table creation, partition management, schema evolution, catalog integration).
• Strong Docker skills: multistage builds, DockerCompose testing, routine image security scanning.
• Experience with at least one major dataprocessing engine (Spark, Flink, or Presto/Trino) and its connection to Iceberg tables.
• Proficiency in Python and/or Ansible for automating infrastructure and platform tasks.
• Experience building CI/CD pipelines that include Docker linting, vulnerability scanning, and automated deployment of datapipeline code.
• Familiarity with observability tooling (Prometheus, Grafana, OpenTelemetry, Loki) and ability to create useful alerts and dashboards.
• Ability to respond to incidents, write clear rootcause analysis reports, and contribute to postmortem actions.
• Willingness to participate in an oncall rotation as a firstline responder.
• Availability to work onsite in NewJersey for the initial assignment and relocate to Dallas by October2026.

Preferred Qualifications

• Experience with cloudnative data services on AWS, Azure, or GCP (EMR, Dataproc, Synapse, etc.).
• Familiarity with other lakehouse formats such as DeltaLake or ApacheHudi and ability to evaluate tradeoffs against Iceberg.
• Knowledge of streaming platforms (Kafka, Pulsar, Kinesis) and realtime processing patterns.
• Relevant certifications (Databricks Lakehouse Associate, Google Professional Data Engineer, AWS Certified Data Analytics - Specialty, etc.).
• Background supporting data platforms in regulated industries (pharma, finance, healthcare) and understanding of associated compliance frameworks.

We're committed to creating a workplace where employees feel valued, supported, and empowered to grow. Our team benefits from competitive compensation, comprehensive health and wellness offerings, and opportunities for professional development. We are proud to be an equal opportunity employer and make all employment decisions without regard to race, color, religion, sex (including pregnancy, sexual orientation, and gender identity), national origin, age, disability, genetic information, veteran status, or any other legally protected status.

We comply with all applicable federal, state, and local employment laws, including those related to fair hiring practices, pay transparency, workplace safety, and reasonable accommodations. We are dedicated to maintaining an inclusive environment where everyone has the opportunity to succeed and contribute meaningfully.