Senior Data Engineer Investment Banking / Risk

Job Title: Senior Data Engineer Investment Banking / Risk
Location: New York, NY
Type: W2 role
Domain: Capital Markets Investment Banking & Risk

Role Summary

We are seeking a Senior Data Engineer to design, build, and optimize large-scale data pipelines supporting Investment Banking and Risk functions. The ideal candidate combines deep hands-on engineering expertise in the Azure Databricks ecosystem with a strong understanding of financial risk data (market risk, credit risk, counterparty risk, regulatory reporting). This role partners closely with quants, risk analysts, and front-office stakeholders to deliver trusted, performant, and audit-ready data products.

Key Responsibilities

• Design and build scalable ETL/ELT pipelines on Azure Databricks using PySpark and Spark SQL to ingest, transform, and curate large volumes of trade, position, market, and reference data.
• Develop and maintain the Medallion (Bronze/Silver/Gold) architecture on Delta Lake, ensuring data quality, lineage, and reconciliation across risk and finance datasets.
• Translate risk and regulatory requirements (e.g., Basel, FRTB, CCAR, VaR, PFE, stress testing) into robust data models and engineering solutions.
• Optimize Spark jobs for performance and cost partitioning, caching, broadcast joins, Z-ordering, and cluster tuning.
• Build and orchestrate workflows using Databricks Workflows / Azure Data Factory, integrating with ADLS Gen2, Azure Key Vault, and CI/CD pipelines.
• Implement data quality, validation, and controls frameworks appropriate to a regulated financial environment.
• Collaborate with quants and risk teams to productionize models and analytical datasets.
• Contribute to code reviews, engineering standards, and documentation.

Required Qualifications

• 7 years of data engineering experience, with 3 years in Investment Banking, Capital Markets, or Risk.
• Strong domain knowledge of risk data market risk, credit risk, counterparty risk, P&L, or regulatory/reg reporting.
• Expert-level Python for data engineering (Pandas, PySpark APIs, modular/production-grade code).
• Advanced PySpark and Spark SQL for distributed data processing at scale.
• Hands-on Azure Databricks and Delta Lake experience (notebooks, Unity Catalog, clusters, jobs).
• Strong SQL skills complex queries, window functions, performance tuning.
• Experience with the broader Azure data stack: ADLS Gen2, Azure Data Factory, Key Vault, Synapse (a plus).
• Solid understanding of data modeling (dimensional, normalized), data warehousing, and lakehouse patterns.
• Experience with version control (Git), CI/CD, and Agile delivery.

Preferred / Nice-to-Have

• Databricks certification (Data Engineer Associate/Professional).
• Exposure to trade lifecycle, OTC derivatives, fixed income, or equities data.
• Familiarity with regulatory frameworks (FRTB, Basel III/IV, BCBS 239).
• Experience with streaming (Structured Streaming, Kafka/Event Hubs).
• Knowledge of data governance, lineage, and Unity Catalog access controls.

Education

Bachelor's or Master's in Computer Science, Engineering, Finance, or a related quantitative field.