What are the responsibilities and job description for the Principal Data Engineer position at Haystack?
We're hiring on behalf of a Haystack partner!
The Role
The Role
- Provide thought leadership in designing, developing, and maintaining robust data systems.
- Lead the development of efficient data processing and ensure data availability for analysis and reporting.
- Drive the design and development of data pipelines from various sources to internal databases.
- Influence the construction and optimization of data infrastructure for analysis readiness.
- Develop and guide the implementation of scalable, sustainable, and robust data products and solutions using advanced engineering and cloud technologies.
- Champion best practices and new data framework concepts to promote standard methodologies in data management.
- Minimum of 6 years of relevant work experience; typically reflects 10 years of experience.
- Define long-term technical direction and establish best practices for scalable and maintainable solutions.
- Champion reliability, observability, and performance for high-standard data systems.
- Make sound technical and architectural decisions by evaluating competing priorities like speed, cost, risk, and flexibility.
- Deep expertise in implementing cloud-based data warehouses, data lakes, and open table formats (e.g., Snowflake, AWS).
- Proficiency in data collection, ingestion tools (e.g., Kafka, AWS Glue), and storage formats (e.g., Iceberg, Parquet).
- Experience with data streaming architectures and tools (e.g., Kafka, Flink) and strong background using Spark for data transformation.
- Extensive experience in DevOps practices, including code management, CI/CD, and deployment strategies.
- Competitive salary based on experience.
- Opportunity to work with cutting-edge big data and cloud-based technologies.
- Play a key role in shaping data strategy and infrastructure for a leading organization.
- Influence technical direction and establish best practices in data engineering.