What are the responsibilities and job description for the Big Data Software Engineer - Intelligent DataOps Software Leader position at Andiamo?
Big Data Software Engineer — Batch & OLAP
About The Role
Build robust batch pipelines and analytical stores for large-scale reporting, experimentation, and ML feature generation.
Responsibilities
Talent Partners for the AI Revolution. As a globally recognized staffing and consulting firm, we specialize in placing the top 2% of technology and go-to-market professionals with the world’s largest and most well-known companies.
For over 20 years, we've maintained the status of tier-one vendor for firms such as Palantir, Amazon, Fluidstack, Bloomberg, Relativity Space, Firefly, MasterCard, Visa, Two Sigma, Citadel, as well as other major financial services firms, elite hedge funds, Google-backed tech start-ups, and major software firms.
Our talent solutions include Permanent Placement, Contract Staffing, Executive Search, and Dedicated Recruiting Services (RPO). Find out more at www.andiamogo.com
About The Role
Build robust batch pipelines and analytical stores for large-scale reporting, experimentation, and ML feature generation.
Responsibilities
- Develop cost-efficient batch processing (Spark/Dataproc/EMR) and orchestration (Airflow/Dagster).
- Design lakehouse tables (Iceberg/Delta/Hudi) with compaction, partitioning, and Z-ordering.
- Implement semantic/metrics layers and OLAP acceleration (BigQuery/Snowflake/ClickHouse).
- Enforce governance: lineage, access controls, PII handling, and retention policies.
- 4 years in big data; expert SQL and performance tuning.
- Strong understanding of storage formats, file layout, and query engines.
Talent Partners for the AI Revolution. As a globally recognized staffing and consulting firm, we specialize in placing the top 2% of technology and go-to-market professionals with the world’s largest and most well-known companies.
For over 20 years, we've maintained the status of tier-one vendor for firms such as Palantir, Amazon, Fluidstack, Bloomberg, Relativity Space, Firefly, MasterCard, Visa, Two Sigma, Citadel, as well as other major financial services firms, elite hedge funds, Google-backed tech start-ups, and major software firms.
Our talent solutions include Permanent Placement, Contract Staffing, Executive Search, and Dedicated Recruiting Services (RPO). Find out more at www.andiamogo.com