Senior Data Engineer

The position is part of a data-focused team working on the “T-Bar” (Transaction Books and Records) product, a critical data store for positions, transactions, and tax lots. The team is transitioning from a legacy data center to a cloud-native AWS environment, aiming to modernize infrastructure and processes by next summer.

Senior Engineer

Must-Have Requirements

• Experience with Batch and Streaming Data Processing: Ability to handle intraday use cases with trading partners, including micro-batches.
• CI/CD and Developer Discipline: Proficiency in code commit practices, including tying commits to Jira tickets for traceability and cherry-picking.
• AWS Familiarity: Experience working within AWS environments, particularly with data pipelines and foundational data layers.
• Python: Strong skills in Python for data processing and logic conversion (e.g., rewriting stored procedures from legacy systems).
• Infrastructure as Code (IaC): Experience with Terraform for deploying and managing infrastructure, particularly for carving out separate AWS accounts.
  
  

Nice-to-Have Skills

• Orchestration Tools: Familiarity with Airflow, Dagster, or similar tools for centralized orchestration (current setup includes step functions, Eventbridge, and an in-house eventing system).
• AI/ML Integration: Exposure to AI tools like GitHub Copilot or Cursor for code development, unit test generation, or data quality checks (e.g., anomaly detection).
• Data Governance: Experience with data catalogs, producing/consuming assets, or tools like AWS DataZone for governance and contract-based testing.
• Event-Driven Architecture: Understanding of event-driven systems, as the team aims to move toward this model in the future.
  
  

#DICE