What are the responsibilities and job description for the Airflow Architect Big Data Systems position at LiveMindz?
Job Summary
Need an experienced Airflow Architect to design, architect, and lead enterprise-scale data orchestration solutions for modern big data platforms. The ideal candidate will have deep expertise in Apache Airflow, Big Data technologies, cloud-native architectures, and data engineering best practices to build scalable, reliable, and high-performance data pipelines.
Key Responsibilities
Architect and implement large-scale data orchestration frameworks using Apache Airflow.
Design, develop, and optimize ETL/ELT pipelines for batch and near real-time processing.
Build scalable workflows integrating Hadoop, Spark, Kafka, Databricks, data lakes, and data warehouses.
Define Airflow deployment strategies, DAG standards, security, monitoring, and governance.
Implement workflow scheduling, dependency management, error handling, retries, alerting, and SLA monitoring.
Lead cloud-based data platform implementations on AWS
Optimize pipeline performance, scalability, reliability, and cost efficiency.
Establish CI/CD processes and DevOps practices for Airflow environments.
Collaborate with business stakeholders, data engineers, architects, and platform teams to translate requirements into technical solutions.
Mentor engineering teams on Airflow architecture, data orchestration patterns, and best practices.
Required Skills
Airflow & Data Engineering
Expert knowledge of Apache Airflow DAG design, scheduling, operators, sensors, and executors.
Strong experience in workflow orchestration, metadata management, and pipeline monitoring.
Hands-on expertise in ETL/ELT development and data pipeline automation.
Big Data Technologies
Apache Spark / PySpark
Hadoop ecosystem
Apache Kafka
Data Warehousing concepts
Data Lakes and Lakehouse architectures
NoSQL databases
Programming
Python (advanced)
SQL (advanced)
Shell scripting
Core Java (preferred)
Cloud Platforms
AWS (EMR, Glue, S3, Redshift)
Preferred Qualifications
10 years of experience in data engineering and big data platforms.
3 years of architecture experience designing enterprise data solutions.
Experience with Kubernetes, Docker, and cloud-native deployments.
Knowledge of Data Mesh, Data Fabric, and Modern Data Platforms.
Experience leading architecture reviews and technology strategy discussions.
Success Metrics
Pipeline reliability and SLA adherence
Workflow performance optimization
Reduced operational incidents
Improved data quality and observability
Scalable and reusable orchestration frameworks