Data Engineer/ Abinitio Consultant

Title: Data Engineer/Ab Initio Consultant

Location: Berkeley Heights, NJ (5 Days Onsite)

Fulltime

Job Description:

We are seeking a highly skilled resource to design and implement high-performance, event-driven data pipelines, ensuring low-latency data processing and high availability system for the large credit card processing system. The ideal candidate will work with the Ab Initio ecosystem (GDE, EME, Conduct>It) to build stateful services that ingest, filter, and transform data from sources like Kafka or message queues, pushing updates to dashboards or downstream databases in near-re  al-time.

Key Responsibilities:

·       Create complex Ab Initio continuous flow graphs, including stateful joins, sliding time windows, and aggregations.

·       Implement event-driven data pipelines using Kafka, MQ, and file streams.

·       Ensure the resilience of continuous flows, including checkpointing and recovery, to guarantee "exactly-once" processing.

·       Apply advanced Ab Initio components (e.g., Reformat, Rollup, Join, Partition) to ensure low-latency performance.

·       Proactively monitor live production streams to ensure 24/7 reliability and troubleshooting data issues

·       Develop ETL pipelines for batch and real-time data ingestion and transformation.

·       Implement and ensure data validation, data security, integrity, and compliance across big data platforms.

·       Monitor and troubleshoot performance issues in large-scale clusters.

·       Collaborate with data scientists, analysts, and application teams to deliver high-quality data solutions.

·       Automate workflows and improve operational efficiency using scripting and orchestration tools.

Required Skills & Qualifications:

·       Deep understanding of credit card process system

·       Deep knowledge of GDE (Graphical Development Environment), EME (Enterprise Meta>Environment), Conduct>It, and Continuous Flows.

·       Understanding of Kafka, message queues, and real-time stateful services.

·       Proficiency in Unix/Linux shell scripting, SQL, and database technologies (e.g., Oracle, Teradata).

·       Experience in Java, Scala, Python or Kafka is plus.

·       Familiarity with Linux/Unix environments and shell scripting.

·       Understanding of data security, governance, and compliance standards.

·       Experience with cloud-based big data platforms

·       Exposure to containerization (Docker, Kubernetes) for big data workloads.

·       Knowledge of CI/CD pipelines for data engineering projects.