Staff Data Engineer/ Lead Data Engineer

Position: Staff Data Engineer
Location: Chicago, IL

As a Staff Data Engineer in the Epsilon Attribution Product Development team, you will:
•    Set the technical direction for large-scale data processing solutions using Scala, Spark, SQL, and modern data platform technologies across the attribution platform.
•    Architect and oversee the operation of trusted data processing pipelines that handle advertiser, customer, and measurement datasets within secure cloud environments (AWS, Google Cloud Platform, Azure) and approved data-sharing ecosystems.
•    Collaborate with Product, Data Science, Security, Privacy, and Platform Engineering teams to deliver privacy-preserving attribution, measurement, forecasting, and analytics solutions.
•    Set the architectural direction for batch and streaming data workflows across the platform, including orchestration frameworks and cloud-native data services.
•    Implement data classification, access controls, and privacy-preserving processing techniques to ensure sensitive datasets and identifiers are handled in accordance with security and compliance requirements.
•    Set the architectural direction for clean-room and trusted data-sharing environments across the platform, ensuring only approved aggregated or privacy-protected outputs are made available for downstream consumption.
•    Build observability, monitoring, and operational tooling to ensure reliability, performance, and compliance of data processing platforms.
•    Troubleshoot complex data platform, performance, and pipeline issues across distributed systems.
•    Set technical direction and drive architecture decisions, engineering best practices, and operational excellence across the attribution platform and adjacent data products.
•    Mentor and grow senior and lead engineers, raise the engineering bar, and provide technical leadership across multiple teams, projects, and product areas.
•    Continuously evolve Epsilon''s attribution, measurement, forecasting, and privacy-preserving analytics capabilities, and influence the long-term technology roadmap.
•    Strong written and verbal English communication skills are required.
•    Good understanding of Agile/SCRUM methodologies and experience working within cross-functional product development teams.

Epsilon''s attribution pipelines process sensitive first-party advertiser data and consumer behavioural signals. A key responsibility of this role is ensuring all data processing occurs within controlled, auditable execution boundaries, no PII or proprietary signals leave the secure perimeter unintentionally.

You will be responsible for:
•    Design and implement data processing pipelines within trusted data environments, clean rooms, secure data-sharing platforms, or equivalent privacy-preserving analytics environments across AWS, Google Cloud Platform, Azure, and on-premises infrastructure.
•    Implement access controls, data classification policies, lineage tracking, and governance controls to ensure PII, PCI-scoped data, customer identifiers, and advertiser-confidential signals are processed only within approved secure environments.
•    Collaborate with Security, Privacy, and Compliance teams to define and maintain data handling standards, ensuring sensitive datasets and raw identifiers remain within approved trust boundaries.
•    Design data flows that enforce privacy-preserving principles, ensuring only aggregated, anonymised, tokenised, or otherwise approved outputs may leave trusted processing environments.
•    Build observability, monitoring, and alerting capabilities to detect anomalous data movement, policy violations, and potential data leakage events.
•    Apply privacy-preserving computation techniques where outputs must cross trust boundaries for downstream analytics and reporting, including:
•    Aggregation before export
•    Pseudonymisation and tokenisation
•    Differential privacy concepts and controls
•    Privacy-aware reporting and measurement
•    Implement encryption, key management, and secure data handling practices using cloud-native security and governance services.
•    Document trust boundaries, data contracts, lineage, and permitted data movement between systems and security zones.
•    Work closely with Security and Privacy teams to support audits, compliance requirements, governance reviews, and secure data-sharing initiatives.
•    Participate in architecture and design reviews for new data products, ensuring data governance, privacy, lineage, and trust-boundary requirements are incorporated from the outset.
•    Contribute to engineering standards and best practices for secure data processing, privacy-preserving analytics, and trusted data platform operations.

What you''ll need
Core technical skills
•    10 years of Data Engineering experience with deep Scala programming expertise and extensive Apache Spark experience for large-scale distributed data processing on AWS and/or Google Cloud Platform.
•    Strong Python development skills for data pipelines, platform tooling, automation, and infrastructure modules .
•    Advanced SQL skills across relational databases, cloud data warehouses, and lakehouse platforms; experience handling TB-scale datasets .
•    Experience designing, building, and maintaining batch and streaming data pipelines.
•    Strong understanding of data warehousing, dimensional modelling, data quality, partitioning, and performance optimisation.
•    Experience with distributed data processing and modern lakehouse architectures (Databricks, Delta Lake, Apache Spark, or equivalent) .
•    Experience building and operating distributed data platforms at scale.
•    Experience with workflow orchestration platforms such as Airflow, Databricks Workflows, AWS Step Functions, or equivalent DAG-based systems.
•    Git or equivalent source control; unit, integration, and automated testing frameworks 
•    Cloud-native development experience across AWS and/or Google Cloud Platform.
•    Strong software engineering practices including CI/CD, code reviews, observability, and production support .
•    Proven ability to set technical direction across multiple teams, mentor senior and lead engineers, drive engineering standards, and consistently deliver complex, cross-cutting initiatives.

Trusted environment execution skills (required)
•    Experience designing and operating data pipelines within trusted data environments, clean rooms, secure data-sharing platforms, or equivalent privacy-preserving analytics environments
•    Experience working with sensitive datasets containing PII, customer identifiers, advertiser data, or regulated information
•    Experience implementing fine-grained access controls, data governance policies, and policy-based enforcement for sensitive datasets
•    Working knowledge of data classification frameworks including PII, PCI, regulated data, and sensitivity-tier models
•    Familiarity with privacy-preserving data processing techniques including tokenisation, pseudonymisation, aggregation-before-export, and differential privacy concepts
•    Experience building or supporting clean-room, measurement, attribution, audience analytics, partner data-sharing, or privacy-preserving reporting solutions
•    Experience with data lineage and governance tooling (Unity Catalog, AWS Glue Data Catalog, Apache Atlas, OpenLineage, or equivalent) for auditability and compliance
•    Understanding of trust boundaries, secure data-sharing patterns, and zero-trust data architecture principles
•    Experience documenting data contracts, data flows, lineage, and permitted movement of data between security zones and business domains
•    Experience with encryption, key management, and secure handling of sensitive data using cloud-native security services
•    Experience designing observability, monitoring, and alerting controls to detect anomalous data movement, policy violations, and potential data leakage events
•    Experience working in environments where only aggregated, anonymised, tokenised, or privacy-protected outputs may leave trusted processing environments
•    Strong understanding of cloud-native security and governance 

Good to have
•    Databricks (Delta Lake, Unity Catalog, Databricks Workflows) 
•    AWS Clean Rooms or equivalent privacy-enhancing technologies 
•    Experience building advertising measurement, attribution, audience activation, retail media, or partner data-sharing platforms 
•    Experience working with Security, Privacy, Risk, or Compliance teams in regulated environments 
•    ELK Stack, Grafana, OpenTelemetry, or equivalent observability platforms 
•    Docker and Kubernetes 
•    Security architecture, threat modelling, and secure design reviews

Thanks,
Rohit Chauhan