What are the responsibilities and job description for the Senior Event Streaming Engineer position at Bdiplus?
Overview
BDIPlus is seeking a Senior Event Streaming Engineer to support a Fortune 100 financial services client’s enterprise data and AI transformation. You will help build an event-driven intelligence platform.
This role sits at the intersection of real-time data engineering, streaming architecture, and AI enablement. You’ll design scalable pipelines that convert raw system activity into high-value, real-time insights used by analytics platforms and AI agents. Your work will directly influence how the business understands and engages with its customers.
What You’ll Do
- Instrument administration systems to capture and emit key customer lifecycle events into Apache Kafka
- Design and build Kafka-to-Iceberg data pipelines using S3 Sink Connector with exactly once delivery guarantees
- Develop real-time behavioral aggregates (7/30/90-day windows) using Kafka Streams with stateful processing and windowed joins
- Configure and manage Schema Registry (Confluent or AWS Glue), enforcing BACKWARD_TRANSITIVE compatibility
- Implement resilient DLQ (Dead Letter Queue) patterns with ordering-safe replay and escalation workflows
- Build and maintain observability frameworks (ELK, Prometheus) for pipeline health, consumer lag, and alerting
- Partner with AI/Agent engineering teams to deliver production-ready streaming data via MCP endpoints
- Develop CDC pipelines for legacy and modern systems (mainframe, AIX/AS400, and policy admin platforms)
- Participate in client demos and contribute to knowledge transfer with internal client teams
- Leverage AI-assisted development tools (Claude Code / Codex) to accelerate delivery
Required Qualifications
- 5 years of hands-on experience with Apache Kafka (topic design, partitioning, scaling, consumer group management)
- Strong expertise in Kafka Streams (stateful processing, windowing, exactly once semantics)
- Experience with Schema Registry (Confluent or AWS Glue) and schema evolution (Avro/Protobuf)
- Proven experience with Kafka Connect and S3 Sink Connector; familiarity with Lakehouse patterns (Apache Iceberg preferred)
- Experience with monitoring and observability (JMX, consumer lag analysis, ELK dashboards)
- Hands-on experience with CDC tools (e.g., Debezium, Informatica IDMC CDC, or equivalent)
- Strong programming skills in SQL and at least one of: Python, Java, or Scala
- Familiarity with AWS services (S3, Glue, Athena, Step Functions)
- Comfort working with AI-assisted development tools (Claude Code experience preferred; training available)
Preferred Qualifications
- Experience working with mainframe data (COBOL copybooks, VSAM, AIX/AS400)
- Background in financial services or insurance
- Hands-on experience with Apache Iceberg (partitioning, schema evolution)
- Familiarity with Denodo or semantic data layer tools
- Experience with infrastructure-as-code (Terraform or CloudFormation)
- Consulting or client-facing experience, including knowledge transfer responsibilities
About BDIPlus
BDIPlus is a data and AI consulting firm with deep experience delivering for Fortune 500 clients, including American Express and Morgan Stanley. We specialize in enterprise data products, identity resolution, semantic layers, and event-driven architecture.
Our engineers are AI-native, leveraging tools like Claude Code from day one to deliver faster, smarter, and more scalable solutions than traditional consulting teams.