Senior Databricks Tech Lead / Architect

Senior Databricks Tech Lead / Architect (Hands-On) — Onsite

Position Overview

We are seeking a Senior Databricks Tech Lead / Architect for an onsite engagement. This is NOT a paper architect role. The ideal candidate must be a hands-on coding practitioner who can architect scalable data solutions, write and review production-quality code, and provide technical direction to an offshore development team. You will be the primary technical authority for our Databricks platform and own both delivery and design quality. This role goes beyond traditional data architecture. We are making significant enterprise investments in AI capabilities and expect this hire to be at the forefront of that evolution. The right candidate will help define and build an intelligence layer on top of our data assets — enabling advanced analytics, AI-driven insights, and future-ready data products. We are looking for someone who actively embraces modern AI tools, coding assistants, and LLMs in their own workflow, and who can articulate and demonstrate how these technologies create measurable business value.

Key Responsibilities

● Design and implement end-to-end data lakehouse solutions on Databricks (Delta Lake, Unity Catalog, Workflows)

● Write hands-on production-quality PySpark, Python, and SQL code — not just review or approve

● Architect medallion (Bronze/Silver/Gold) data pipelines and ensure they meet performance and reliability standards

● Lead and direct offshore/nearshore development teams: daily standups, code reviews, sprint planning, and technical mentorship

● Define and enforce coding standards, branching strategies, and CI/CD practices for the data engineering team

● Collaborate with business stakeholders, data architects, and product owners to translate requirements into technical designs

● Optimize Spark jobs, cluster configurations, and cost management across cloud environments (AWS, Azure, or Google Cloud Platform)

● Integrate Databricks pipelines with cloud-native services across hyperscalers — including AWS (Glue, S3, Kinesis, Lambda), Azure (ADF, Event Hubs, ADLS), or Google Cloud Platform (Dataflow, BigQuery, Pub/Sub) — as well as REST APIs and messaging systems

● Drive best practices in data quality, data governance, and metadata management

● Proactively identify technical debt and lead remediation efforts

● Serve as the go-to escalation point for all Databricks technical issues within the engagement

● Design and build an enterprise intelligence layer on top of data assets — enabling AI-driven insights, semantic data products, and advanced analytics consumption patterns

● Evaluate and integrate enterprise AI platforms, LLM frameworks, and vector/embedding stores (e.g., Databricks Vector Search, AWS Bedrock, LangChain) into the data architecture

● Champion the use of AI coding assistants and modern developer productivity tools across the engineering team

● Translate AI and data capabilities into clear business value narratives for executive and business stakeholder audiences

Required Qualifications

● 8 years of overall data engineering experience; 4 years specifically on Databricks platform

● Deep hands-on expertise with PySpark, Spark SQL, Delta Lake, and Databricks Workflows

● Proven experience architecting large-scale data pipelines on at least one major hyperscaler — AWS (S3, Glue, Redshift, Kinesis, EMR), Azure (ADF, ADLS, Synapse, Event Hubs), or Google Cloud Platform (Dataflow, BigQuery, Pub/Sub)

● Demonstrated experience leading and directing offshore or distributed engineering teams

● Proficiency in Python and SQL — must be able to produce and review code, not just advise

● Strong grasp of data modeling, dimensional modeling, and lakehouse design patterns

● Experience with CI/CD tools (GitHub Actions, Azure DevOps, AWS CodePipeline) and infrastructure-as-code (Terraform, AWS CDK, ARM/Bicep)

● Demonstrated awareness of and curiosity about the evolving AI landscape — including LLMs, generative AI, AI coding assistants, and enterprise AI platforms

● Practical, hands-on experience using AI tools (e.g., GitHub Copilot, ChatGPT, Claude, Cursor, or similar) to improve personal and team productivity

● Ability to bridge data architecture and AI/ML capabilities — understanding how data assets enable AI-driven products and insights

● Databricks Certified Associate Developer or Professional certification is a strong plus

● Excellent communication and stakeholder management skills for both technical and business audiences

Enterprise AI & Intelligence Layer

● This role is central to our enterprise AI strategy. We expect the Architect to actively shape how AI capabilities are integrated into our data platform — not just support it from the sidelines.

● The candidate should be able to design the architecture for an intelligence layer that sits on top of Gold-tier data assets — enabling use cases such as AI-driven reporting, natural language querying, predictive analytics, and LLM-powered data products

● Candidates must demonstrate active, personal engagement with modern AI tooling — we want to see real examples of how they use AI assistants and LLMs in their day-to-day engineering work

● The ability to speak credibly to both engineers and business leaders about where AI is heading, and how the data platform positions the organization to take advantage of it, is a key differentiator for this role

● Familiarity with responsible AI principles, including data privacy, model governance, and bias awareness, is expected at this seniority level

Preferred / Nice-to-Have Skills

● Experience with Unity Catalog, Delta Sharing, and Databricks SQL Warehouses

● Hands-on experience with Databricks AI/BI, MLflow, and Databricks Model Serving

● Familiarity with LLM orchestration frameworks such as LangChain, LlamaIndex, or Semantic Kernel

● Experience with vector databases or embedding stores (e.g., Databricks Vector Search, Pinecone, OpenSearch)

● Experience integrating Databricks with cloud-native data services across AWS (Glue, Lake Formation, Redshift), Azure (ADF, Synapse, ADLS), or Google Cloud Platform (Dataflow, BigQuery)

● Familiarity with cloud AI/ML services: AWS Bedrock/SageMaker, Azure OpenAI/ML Studio, or Google Cloud Platform Vertex AI

● Prior consulting or client-facing delivery experience with a track record of translating technical capabilities into business outcomes