Data Engineer - Global Hedge Fund - 300k+

This is a data engineering role inside a global hedge fund where the data platform is not a support function, it is the thing the investment process runs on. The role owns pipelines that feed research, risk, and portfolio construction, which means the engineer sits in constant contact with the people actually making decisions with the output. Databricks and Spark are the core of the stack, and the work spans ingestion from vendors and exchanges through to the curated datasets researchers query every morning. If a pipeline fails silently and a dataset lands wrong, a portfolio manager makes a call on bad numbers before anyone notices. That is the standard the role is held to.

What You'll Do

• Design, build, and maintain large scale batch and streaming pipelines in Spark on Databricks
• Own the full lifecycle of datasets, from vendor onboarding and validation through to the models and tables researchers consume
• Work directly with researchers, portfolio managers, and risk to translate vague data questions into concrete engineering work
• Build the monitoring, testing, and data quality controls that catch problems before consumers do
• Improve performance and cost of existing Spark workloads, and make decisions about how the platform evolves
• Document and explain your work to technical and non technical audiences

Must-haves

• 2 to 10 years of data engineering experience
• Strong production experience with Databricks and Spark, including performance tuning
• Strong Python and SQL
• Demonstrated ability to communicate clearly with non engineers, defend technical decisions in plain language, and manage stakeholders directly. This is non-negotiable for the role, the engineer who cannot hold a conversation with a portfolio manager will not succeed here
• Experience with cloud data infrastructure (AWS, Azure, or GCP)

Nice-to-haves

• Exposure to financial or market data (tick data, reference data, corporate actions)
• Delta Lake, Unity Catalog, or lakehouse architecture experience
• Orchestration tooling (Airflow, Dagster, Databricks Workflows)
• Infrastructure as code and CI/CD for data pipelines
• Experience in a buy side or trading environment

Why This Role

The data platform here has direct, traceable impact on investment performance, and the engineers who build it are known by name to the people using it. The team is small enough that individual decisions shape the architecture rather than disappearing into a backlog, and the pace reflects a business where data problems are urgent by default. For a data engineer who is tired of building pipelines for people they never meet, and who wants their technical judgment to be visible to the desk, this is that.

Who should apply

Strong data engineers from top technology companies are actively encouraged to apply. No finance background required, but genuine curiosity about markets is expected.