Senior Data Architect

We are looking for a Senior Data Engineer or Architect for our Client's data platform modernization and consolidation efforts: consolidating fragmented data sources into a Databricks-based Lakehouse architecture.

As the Senior Data Engineer / Data Architect, you will define the end-to-end data architecture, own the semantic and dimensional data models that power investment decision-making, and establish data governance and quality standards across the organization. You will also build Python-native analytics and visualization applications, enabling portfolio managers, risk teams, and compliance stakeholders to interact directly with high-quality data.

Key Responsibilities

Data Platform Architecture

• Define and govern the overall Databricks Lakehouse architecture, including bronze/silver/gold layers, Delta Lake table design, Unity Catalog structure, and cluster policies
• Design scalable, performant architectures aligned with analytical and regulatory needs

Data Modeling & Semantic Layer

• Design and maintain dimensional data models covering portfolio, positions, security master, counterparties, NAV, and performance attribution
• Own the dbt Semantic layer, including modeling standards, testing frameworks, documentation, and code review for downstream models

Analytics & Data Applications

• Build and maintain Python-based analytics dashboards and internal tools using Plotly /Dash or Streamlit
• Lead all Python-native visualization and analytical app development (Power BI is handled by existing staff)

Performance & Regulatory Reporting

• Own the GIPS-compliant performance reporting data mart
• Automate regulatory reporting outputs, including 13F and CFTC position reports

Data Governance & Quality

• Establish and enforce the firm’s data governance framework: Unity Catalog permissions, data cataloging, lineage, and data dictionaries
• Define and monitor data quality standards and SLAs across all pipelines

Technical Leadership

• Act as technical lead and mentor for the Data Engineer, providing code reviews, architectural guidance, and engineering standards
• Evaluate and introduce new tooling as the platform evolves (e.g., MLflow, Databricks Feature Store)

LLM & Advanced Analytics

• Own the LangChain and LangGraph stack, building LLM-powered agents, RAG pipelines, and multi-step workflows for internal investment research and operations
• Instrument and monitor LLM applications using LangSmith for tracing, evaluation, and production reliability

Required Qualifications

• 6 years of data engineering experience, including 2 years in a senior, lead, or architect role
• Deep expertise in Databricks, including Unity Catalog, Delta Lake internals, SQL warehouse optimization, Photon engine, and cluster autoscaling
• Hands-on production experience with LangChain and LangGraph, building LLM agents, RAG pipelines, and stateful workflows
• Strong Python skills (pandas, PySpark) and experience with Streamlit or Plotly/Dash
• Expert-level SQL, including complex modeling, query optimization, and execution planning
• Extensive dbt experience (advanced testing, macros, packages, semantic layer design)
• Proven experience designing medallion lakehouse architectures
• Financial services data experience (portfolio, position, trade, or market data),
• Demonstrated ability to implement data governance, cataloging, lineage, access control, and data quality frameworks
• Experience with Bloomberg Data License, FactSet, or custodian feeds (BNY Mellon, State Street)
• Knowledge of GIPS performance standards and composite construction
• Familiarity with regulatory reporting (13F, CFTC, AIFMD, Form PF)
• Experience with LangSmith for LLM observability and evaluation
• Exposure to Databricks MLflow, Feature Store, Kafka, or Delta Live Tables

The anticipated salary range for this position is $150-200K. Actual salary will be based on a variety of factors including relevant experience, knowledge, skills and other factors permitted by law. A range of medical, dental, vision, retirement, paid time off, bonus and/or other benefits are available.