What are the responsibilities and job description for the Data Engineer - Airflow, dbt, Kubernetes/OpenShift - Jersey City, NJ position at Expert Technology Services?
Job Summary (Senior Data Engineer):
- Lead the design, development, and maintenance of complex data pipelines using Apache Airflow and dbt Core for enterprise-scale financial and accounting platforms.
- Build, operate, and optimize scalable, high-volume data pipelines supporting system migrations and large data processing workloads.
- Develop and optimize Airflow DAGs, manage workflow orchestration, and ensure high performance, reliability, and error handling.
- Architect, implement, and maintain robust dbt Core models, including staging, intermediate, and mart layers while following analytics engineering best practices.
- Deploy and manage data workloads on cloud-native container platforms (Kubernetes/OpenShift), ensuring efficient resource allocation and scalability.
- Monitor, troubleshoot, and tune end-to-end pipeline and platform performance, addressing bottlenecks and ensuring high availability and resiliency.
- Implement observability solutions (logs, metrics, alerts) for proactive monitoring and rapid issue resolution.
- Collaborate with data architects, platform engineers, and business stakeholders to support financial reporting, accounting, and regulatory data use cases.
- Enforce data engineering standards, security, and governance policies across data platforms and workflows.
- Leverage expertise in Python, SQL, distributed workload management, containerization, and CI/CD to deliver robust and scalable data solutions.
- Preferred: Experience with financial services/accounting platforms, enterprise system migrations, and data warehouses (e.g., Oracle).
- Lead the design, development, and maintenance of complex data pipelines using Apache Airflow and dbt Core for enterprise-scale financial and accounting platforms.
- Build, operate, and optimize scalable, high-volume data pipelines supporting system migrations and large data processing workloads.
- Develop and optimize Airflow DAGs, manage workflow orchestration, and ensure high performance, reliability, and error handling.
- Architect, implement, and maintain robust dbt Core models, including staging, intermediate, and mart layers while following analytics engineering best practices.
- Deploy and manage data workloads on cloud-native container platforms (Kubernetes/OpenShift), ensuring efficient resource allocation and scalability.
- Monitor, troubleshoot, and tune end-to-end pipeline and platform performance, addressing bottlenecks and ensuring high availability and resiliency.
- Implement observability solutions (logs, metrics, alerts) for proactive monitoring and rapid issue resolution.
- Collaborate with data architects, platform engineers, and business stakeholders to support financial reporting, accounting, and regulatory data use cases.
- Enforce data engineering standards, security, and governance policies across data platforms and workflows.
- Leverage expertise in Python, SQL, distributed workload management, containerization, and CI/CD to deliver robust and scalable data solutions.
- Preferred: Experience with financial services/accounting platforms, enterprise system migrations, and data warehouses (e.g., Oracle).