What are the responsibilities and job description for the Senior Data Engineer position at Gunvor Group?
Job Title:
Senior Data Engineer
Contract Type:
Time Type:
Job Description:
Main Responsibilities
Data Engineering Leadership
Our people make all the difference in our success.
Senior Data Engineer
Contract Type:
Time Type:
Job Description:
Main Responsibilities
Data Engineering Leadership
- Contribute to the definition of the target data architecture (lake/lakehouse, streaming, event-driven) and technology choices for high-volume time-series, market, and transactional data.
- Build and operate end-to-end pipelines (ingest → quality → transform → serve) with strong SLAs/SLOs, lineage, and observability.
- Establish coding, testing, CI/CD, and infrastructure-as-code standards; drive adoption of data governance, cataloging, and access controls.
- Own capacity, performance, reliability, and cost efficiency of data platforms across environments (dev/test/prod).
- Ensure data quality, lineage, and observability across all layers of the data stack.
- Partner with trading desks, quantitative teams, and risk functions to translate business needs into data solutions that enhance decision-making and operational efficiency.
- Act as a senior liaison between engineering and business stakeholders, ensuring alignment on data priorities and delivery timelines.
- Prioritize a value-based backlog (e.g., faster close/settlement, improved forecast accuracy, reduced balancing penalties) and measure business impact.
- Align data models and domain ownership with business processes (bids/offers, nominations, positions, exposures,outages).
- Coordinate with Cybersecurity, Compliance, and Legal on sector-specific controls (e.g., REMIT/NERC-CIP considerations, data retention, segregation).
- Incubate and industrialize data products: curated marts, feature stores, real-time decision APIs, and event streams for forecasting and optimization.
- Introduce modern patterns (CDC, schema evolution, Delta/Iceberg, stream–batch unification) to improve freshness and resilience.
- Evaluate and integrate external data (weather, fundamentals, congestion, capacity postings), internal and external vendor systems (ETRM) safely and at scale.
- Collaborate with quantitative analysts to productionize ML pipelines (forecasting load/renewables, anomaly detection, etc.. ) with monitoring and rollback.
- Lead incident reviews and architectural forums; provide pragmatic guidance on trade-offs (latency vs. cost, simplicity vs. flexibility).
- Develop growth paths and learning plans focused on energy domain fluency and modern data engineering practices.
- Implement robust monitoring/alerting, runbooks, and on-call rotations; drive MTTR down and availability up for critical data services.
- Enforce data quality contracts (SLAs/SLOs), lineage, and reconciliation for market submissions, settlements, and reporting.
- Optimize cloud spend and storage/compute footprints; plan capacity for market events and seasonal peaks.
- Ensure security and compliance by design: least-privilege access, secrets management, encryption, auditability, and disaster recovery testing.
- Master’s or Bachelor’s degree in Computer Science, Data Engineering, Applied Mathematics, or a related technical field.
- 5 years of experience in data engineering, with at least 3 years in a senior role.
- Proven experience in the energy trading sector, ideally with exposure to Natural Gas and Power markets, balancing mechanisms, and regulatory frameworks (e.g., REMIT, EMIR).
- Azure: ADLS Gen2, Event Hubs, Synapse Analytics, Azure Databricks (Spark), Azure Functions, Azure Data
- Factory/Databricks Workflows, Key Vault, Azure Monitoring/Log Analytics; IaC with Terraform/Bicep; CI/CD with Azure DevOps or GitHub Actions.
- Snowflake (on Azure or multi-cloud): Warehousing design, Streams & Tasks, Snowpipe/Snowpipe Streaming, Time
- Travel & Fail-safe, RBAC & row/column security, external tables over ADLS, performance tuning & cost governance.
- Programming & Engineering Practices: Strong OOP in Python and Java/Scala; SDLC leadership, DevOps mindset, TDD/BDD, code reviews, automated testing (unit/integration/contract), packaging and dependency management, API design (REST/gRPC).
- Web & Data Acquisition: Robust web scraping and ingestion with Scrapy, Requests, Playwright/Selenium; scheduling, retries/exponential backoff, change-data capture; ethical/legal collection practices (robots.txt, terms).
- Orchestration & Quality: Airflow/ADF/Databricks Jobs, data contracts, Great Expectations (or similar), lineage/catalog (e.g., Purview), metrics/observability (Prometheus/Grafana/Application Insights).
- Dataframe oriented programming: pandas, spark/snowpark dataframes, SQL data transformation
- Additional Skills
- Designing low-latency pipelines for sub-second to minute-level telemetry, weather and market data; tuning Spark
- Structured Streaming/Flink/Kafka Streams.
- Quality & Reconciliation for telemetry and market submissions (gap fill, resampling, deduplication, anomaly detection, schema evolution).
- Serving Patterns: time-series stores and query layers (e.g., Delta Lake over ADLS, Iceberg, materialized views in Snowflake), APIs and event streams for downstream consumption.
- English (fluent), any additional language is an asset
Our people make all the difference in our success.