Apache Airflow Engineer

Title: Apache Airflow Engineer
Location: New York City, NY (Hybrid)
Duration: 6 months (possibility of extension)
JD:

Experience:

• 7 years overall for Engineer

Level:

• Senior Consultant / Lead Consultant (Onsite Lead)
• Consultant / Senior Consultant (Offshore Engineer)

Core Focus

• Cybermation to Apache Airflow migration
• Airflow DAG development
• Shell script/API scheduling
• Operational validation
• Runbook creation
• Production support

Job Summary

We are seeking Apache Airflow migration resources to support a client initiative focused on migrating existing Cybermation-scheduled scripts to Apache Airflow. The resources will analyze the current Cybermation schedules, understand the existing shell/API invocation patterns, convert the schedules into Airflow DAGs, validate execution behavior, and support production deployment and handover.

The migration objective is to preserve the existing business execution behavior while improving schedule control, monitoring, logging, retry handling, operational visibility, and supportability through Apache Airflow.
Key Responsibilities

• Analyze existing Cybermation job schedules, dependency patterns, runtime parameters, execution frequency, owners, success/failure behavior, and operational support expectations.
• Review existing shell scripts and API/cURL invocation logic to determine the appropriate Airflow DAG design, task structure, error handling, and retry configuration.
• Design, develop, and maintain Apache Airflow DAGs to orchestrate shell scripts, API calls, and dependent jobs with appropriate scheduling and operational controls.
• Configure Airflow schedules, task dependencies, retries, logging, alerting hooks, variables, connections, parameters, and environment-specific configuration as per client standards.
• Perform dry runs, unit testing, schedule validation, API response validation, failure/retry validation, and production-readiness checks for migrated jobs.
• Create migration mapping, validation evidence, deployment notes, operational runbooks, and support handover documentation.
• Coordinate with client SMEs, application owners, infrastructure teams, and release teams to support deployment, issue triage, initial run monitoring, and hypercare.

Required Skills

• Hands-on experience in Apache Airflow DAG development, scheduling, dependency management, retries, operators, variables, connections, logging, monitoring, and migration from legacy schedulers.
• Experience or strong understanding of legacy enterprise schedulers such as Cybermation, Control-M, Autosys, Tidal, UC4, cron, or similar tools, with the ability to convert schedules into Airflow workflows.
• Strong Linux/Unix experience, shell scripting, cURL/API execution, environment variables, job logs, exit codes, file permissions, and scheduler operational patterns.
• Good Python scripting skills for Airflow DAG development, parameterization, lightweight automation, input validation, logging, and exception handling.
• Ability to work with REST APIs, cURL-based calls, authentication patterns, status validation, response handling, retries, and audit logging from Airflow workflows.
• Experience with Git/source control, deployment coordination, environment configuration, production support, incident triage, runbooks, and defect resolution.
• Strong communication skills, documentation discipline, ability to work with client stakeholders, and ability to operate effectively in an onsite/offshore delivery model.

Preferred Qualifications

• Prior experience migrating jobs from Cybermation or equivalent enterprise schedulers to Apache Airflow.
• Experience deploying or operating Airflow in cloud or containerized environments is preferred.
• Exposure to AWS services such as EKS, S3, CloudWatch, Lambda, Step Functions, IAM, Secrets Manager, or EventBridge is a plus.
• Experience supporting financial services, capital markets, or other regulated enterprise environments is preferred.
• Familiarity with ITSM processes, release governance, change management, and production support procedures.

Education & Experience

• Bachelor's degree or equivalent experience in Computer Science, Information Technology, Engineering, or a related discipline.
• 7 years of overall engineering experience with hands-on Airflow, Python, and Linux/Shell exposure.
• Strong practical understanding of job scheduling, batch/workflow orchestration, production readiness, and operational support processes.

Role Deliverables

• Migration inventory and mapping of Cybermation schedules to target Airflow DAGs.
• Developed and configured Airflow DAGs for in-scope shell script and API/cURL-based workflows.
• Validated execution evidence covering dry runs, schedule validation, API response checks, failure/retry validation, and defect remediation.
• Deployment notes, operational runbooks, manual re-run steps, monitoring instructions, and production support handover documentation.
• Post-deployment validation and hypercare support for migrated workflows.

Tools & Technologies

Apache Airflow, Python, Linux/Unix, Shell Scripting, cURL, REST APIs, Git, CI/CD basics, YAML/JSON configuration, scheduler migration, job monitoring, logging, retry handling, production support, runbook documentation, and optional AWS services such as EKS, S3, CloudWatch, Lambda, IAM, Secrets Manager, EventBridge, and Step Functions.