What are the responsibilities and job description for the ETRM Databricks Data Engineer position at Programmers.io?
Job Description:
Position: ETRM Databricks Data Engineer
Location: Houston, Texas (5 days on-site)
Hire Type: Full Time
ETRM Databricks Data Engineer to design, develop, and maintain scalable data solutions that support business intelligence, analytics, and data-driven decision-making with understanding of Data Domain & Data Assets in ETRM Domain. The ideal candidate will have strong expertise in Databricks, Apache Spark, Azure Cloud platforms, and modern data engineering practices.
Key Responsibilities:
- Design, develop, and optimize data pipelines using Databricks & ADF.
- Must possess good understanding of Trading Domain knowledge.
- Build and maintain scalable ETL/ELT processes for structured and unstructured data.
- Integrate data from multiple sources into enterprise data platforms.
- Develop and manage data lakes, lakehouses, and data warehouses.
- Collaborate with data scientists, analysts, and business stakeholders to understand data requirements.
- Ensure data quality, integrity, security, and governance across data platforms.
- Monitor and troubleshoot data processing jobs and optimize performance.
- Implement CI/CD practices and automate data engineering workflows.
- Create and maintain technical documentation, data models, and process flows.
- Support cloud-based data solutions and migration initiatives.
Required Qualifications:
- Hands-on experience with Databricks and Azure.
- Strong proficiency in Python, SQL, and PySpark.
- Experience with cloud platforms such as Azure
- Knowledge of data warehousing, data modeling, and ETL/ELT concepts.
- Experience with version control tools such as Git.
- Strong analytical, problem-solving, and communication skills.
- Trading Domain experience related for Crude, Refined Products, Natural gas is a must.
Preferred Qualifications:
- Experience with Azure Data Factory, Azure Synapse Analytics, Delta Lake, or similar technologies.
- Knowledge of streaming data processing and real-time analytics.
- Familiarity with DevOps practices and CI/CD pipelines.
- Databricks, Azure, or related cloud certifications.