What are the responsibilities and job description for the Data Engineer position at Scale.jobs?
About The Role
The role focuses on the architecture and implementation of scalable data infrastructure that powers both real-time product features and high-stakes business intelligence. This position sits at the intersection of software engineering and data science, ensuring that datasets are performant, reliable, and accessible for downstream consumers.
The team manages high-volume data ingestion from disparate sources, transforming raw events into structured, governed schemas. This role is critical for maintaining the reliability of data products while optimizing the cost and latency of the entire data lifecycle.
Key Responsibilities
The role focuses on the architecture and implementation of scalable data infrastructure that powers both real-time product features and high-stakes business intelligence. This position sits at the intersection of software engineering and data science, ensuring that datasets are performant, reliable, and accessible for downstream consumers.
The team manages high-volume data ingestion from disparate sources, transforming raw events into structured, governed schemas. This role is critical for maintaining the reliability of data products while optimizing the cost and latency of the entire data lifecycle.
Key Responsibilities
- Design and build robust, scalable ELT/ETL pipelines using Python and SQL to ingest data from internal microservices and third-party APIs
- Develop and maintain complex data models in Snowflake or BigQuery using dbt, ensuring high performance for analytical queries
- Orchestrate batch and streaming workflows using Airflow, Prefect, or Dagster to manage task dependencies and ensure data freshness
- Implement data quality monitoring and automated testing frameworks to identify and resolve upstream data drift and schema changes
- Optimize warehouse compute usage and storage strategies to balance performance requirements with operational cost efficiency
- Collaborate with backend engineers to define event logging schemas and ensure data consistency across the distributed system
- 3–6 years of experience in data engineering, with a proven track record of managing production-grade data pipelines
- Expert-level proficiency in Python and advanced SQL, including window functions, query optimization, and complex joins
- Hands-on experience with modern cloud data warehouses such as Snowflake, Redshift, or BigQuery
- Proficiency with data transformation tools and orchestrators, specifically dbt and Apache Airflow
- Strong understanding of data modeling principles, including Star Schema, Snowflake Schema, and Data Vault 2.0
- BS or MS in Computer Science, Engineering, Mathematics, or a related technical field
- Bonus: Experience with streaming technologies like Kafka or Flink, and infrastructure management via Terraform