Databricks Data Engineer

Senior Data Engineer

Join a growing data and engineering team focused on building scalable data products, modernizing data infrastructure, and creating trusted datasets that support analytics, automation, and AI-driven applications. This role combines hands-on engineering with platform ownership, giving you the opportunity to influence how data is collected, transformed, governed, and delivered across the organization.

You'll work with cloud-native technologies, distributed processing frameworks, and modern orchestration tools to build reliable data systems that power critical business and technology initiatives.

What You'll Be Doing

• Design, develop, and maintain scalable data pipelines that collect and process information from APIs, databases, file transfers, and third-party platforms.
• Build and manage data models that support reporting, analytics, machine learning, and operational workloads.
• Create and optimize batch and streaming data workflows using Python, SQL, Spark, Kafka, and related technologies.
• Develop orchestration frameworks using tools such as Prefect or Airflow to automate scheduling, monitoring, recovery processes, and operational workflows.
• Implement cloud-based data solutions using services across AWS, Azure, or GCP, leveraging object storage, distributed compute, and managed data services.
• Manage data lifecycle processes, including ingestion, transformation, validation, cataloging, and delivery.
• Establish strong data quality standards through automated testing, monitoring, observability, and lineage tracking.
• Improve platform performance through query optimization, workload tuning, partitioning strategies, and efficient resource utilization.
• Partner closely with data scientists, software engineers, AI teams, analysts, and business stakeholders to deliver production-ready datasets and data products.
• Support emerging AI and agent-based applications by preparing structured datasets, metadata, and integration layers for downstream consumption.
• Drive engineering best practices around documentation, automation, reliability, security, and maintainability.
• Contribute to architectural decisions and help shape the future direction of the organization's data platform.

Required Experience

• Bachelor's degree in Computer Science, Information Systems, Engineering, or a related technical field.
• 4 years of experience building and supporting enterprise-scale data platforms.
• Strong programming experience with Python and advanced SQL.
• Hands-on experience with Spark, PySpark, Kafka, or other distributed data processing technologies.
• Experience developing ETL and ELT pipelines in cloud-based environments.
• Strong knowledge of modern data warehousing and lakehouse concepts.
• Experience working with Snowflake, including automated ingestion, change data capture, and task orchestration capabilities.
• Experience with Databricks or similar large-scale data processing platforms.
• Familiarity with cloud services across AWS, Azure, or Google Cloud.
• Experience using dbt for transformation and data modeling.
• Knowledge of Git workflows, CI/CD practices, and container technologies such as Docker.
• Strong understanding of data governance, security controls, and access management principles.

Preferred Qualifications

• Experience designing and managing enterprise APIs and data service layers.
• Knowledge of OAuth, JWT authentication, network security, and API governance frameworks.
• Experience building data solutions that support generative AI, LLM, or agent-based applications.
• Familiarity with MCP, LangChain, LlamaIndex, or related AI integration frameworks.
• Experience implementing data observability, monitoring, and quality platforms.
• Understanding of metadata management, lineage tracking, and data catalog solutions.
• Exposure to highly regulated environments with strong data privacy and compliance requirements.
• Experience supporting large-scale analytics, machine learning, or AI workloads in production.

What Success Looks Like

You'll build reliable data systems that teams trust, reduce manual processes through automation, improve data accessibility across the organization, and create a strong foundation for advanced analytics and AI initiatives. Your work will directly influence how data is used to drive decisions, improve efficiency, and support future innovation.