What are the responsibilities and job description for the Senior Data Engineer position at Emergent Professional Resources L.P. (EPR)?
Our Midstream Oil and Gas client is seeking a skilled Data Engineer to design, build, and maintain scalable data infrastructure that supports advanced analytics and business intelligence initiatives. You will work closely with data scientists, analysts, and software engineers to ensure data availability, quality, and reliability across the organization.
Key Responsibilities
- Design, develop, and maintain data pipelines and ETL/ELT processes for collecting, transforming, and storing data from multiple sources
- Build and optimize data models, data warehouses, and data lakes for analytics and reporting.
- Collaborate with stakeholders to define data architecture and ensure data integrity, consistency, and security
- Implement and maintain data quality frameworks, monitoring, and alerting systems.
- Work with cloud platforms (AWS, Azure, or GCP) to deploy scalable data infrastructure
- Support the integration of real-time streaming data using tools like Kafka, Kinesis, or Spark Streaming
- Automate workflows using orchestration tools such as Airflow, Prefect, or Dagster
- Ensure compliance with data governance, security, and privacy standards (e.g., GDPR, HIPAA).
- Troubleshoot data-related issues and optimize performance of queries and pipelines
Required Qualifications
- 7 years of experience as a Data Engineer or similar role
- Strong proficiency in SQL and a programming language such as Python
- Hands-on experience with ETL frameworks, data warehousing, and big data tools (e.g., Spark, Hadoop).
- Experience with cloud-based data solutions such as AWS Redshift, Snowflake, BigQuery, or Azure Synapse.
- Proficiency with data orchestration and workflow automation tools
- Solid understanding of data modeling, data architecture, and API integration.