What are the responsibilities and job description for the Data Engineer position at IT Resources?
Role Overview
We're looking for a Data Engineer with a strong focus on machine learning workflows and logistics data integration. In this role, you'll design and maintain scalable data pipelines, collaborate closely with data scientists, and transform operational data from WMS and ERP systems into actionable insights. If you're passionate about building ML-ready data infrastructure and thrive in cross-functional environments, we’d love to hear from you.
Key Responsibilities
ML-Focused Data Engineering
- Build and optimize data pipelines tailored for machine learning use cases
- Collaborate with data scientists to prepare features, manage data versions, and support model retraining
- Lead initiatives around feature stores, input validation, and monitoring systems
Data Integration from WMS & Operational Systems
- Ingest and transform structured/unstructured data from WMS, ERP, and telemetry platforms
- Model and enrich logistics data for real-time and predictive applications
Pipeline Automation & Orchestration
- Design modular, scalable pipelines using tools like Azure Data Factory, Airflow, or Databricks Workflows
- Ensure data freshness and reliability across batch and streaming environments
Data Governance & Quality
- Implement validation layers, anomaly detection, and reconciliation processes
- Contribute to metadata management, data lineage, and auditability
Cross-Functional Collaboration
- Partner with data scientists, ML engineers, software developers, and warehouse operations teams
- Translate modeling needs into optimized, structured data architecture
Documentation & Mentorship
- Document ML data flows, WMS mappings, and pipeline logic for team-wide clarity
- Mentor junior team members on best practices in ML data engineering
Required Qualifications
Technical Skills
- Strong experience with ML data pipelines, feature engineering, and model lifecycle support
- Proficient in Python, SQL, and tools like dbt, Spark, or Delta Lake
- Deep understanding of data modeling, orchestration, and cloud platforms (Azure, Databricks)
- Familiarity with ML tools like MLflow, TensorFlow Extended (TFX), or similar
- Hands-on experience with WMS or logistics data
Experience
- 5 years in data engineering, including 2 years supporting AI/ML teams
- Proven track record designing production-grade pipelines in cloud environments
- Strong