Databricks SME/Lead

We are looking for an experienced Databricks SME/Lead to guide architecture, design, and implementation across our cloud data engineering initiatives. This senior role requires deep technical expertise, exceptional attention to detail, and the ability to advise engineering teams while also coordinating with senior leadership on high-impact architectural decisions.

Key Responsibilities

• Serve as the Databricks platform SME and trusted technical advisor for engineering teams.
• Create Architecture patterns to leverage Databricks features for Batch, real-time, Streaming & Operational analytics.
• Drive solutions with Databricks to accelerate data engineering productivity, ensure data protection & security, improve performance of jobs & queries.
• Review architecture diagrams, data workflows, pipelines, notebooks, and solution designs for accuracy and best practices.
• Provide corrective guidance, optimization suggestions, and technical leadership on Databricks implementations.
• Function as the bridge between engineering teams and leadership, escalating high-impact decisions when necessary.
• Enforce enterprise standards for architecture, governance, security, and performance across Databricks solutions.
• Mentor teams on Databricks, Spark, Delta Lake, and Lakehouse architectural patterns.
• Collaborate with architects, data engineers, and platform teams on pipeline design, orchestration, cluster strategy, and data optimization.
• Contribute to technical roadmaps, reusable patterns, and long-term platform strategy.
  
  

Required Qualifications

• 8 years in data engineering or architecture roles, with deep Databricks platform expertise.
• Strong proficiency in PySpark, Spark SQL, Delta Lake, Lakehouse architecture, and Databricks Jobs/Workflows.
• Proven ability to review and validate architectural diagrams, solution designs, and platform components.
• Hands-on experience designing, optimizing, and troubleshooting Databricks pipelines and production workloads.
• Strong understanding of data governance, Unity Catalog, cluster management, CI/CD for Databricks, and performance tuning.
• Excellent communication and documentation skills, with the ability to mentor teams and act as a senior escalation point.
• Highly detail-oriented with strong analytical and technical evaluation skills.
  
  

Nice To Have

• Experience in hybrid roles combining technical execution with architectural governance.
• Cloud expertise (Azure, AWS, or GCP) in relation to Databricks deployments.
• Relevant certifications (Databricks, Spark, AWS/Azure/GCP).
  
  

Benefits

• 401(k).
• Dental Insurance.
• Health insurance.
• Vision insurance.
• We are an equal-opportunity employer and value diversity, equality, inclusion, and respect for people.
• The salary will be determined based on several factors, including, but not limited to, location, relevant education, qualifications, experience, technical skills, and business needs.
  
  

Additional Responsibilities

• Participate in OP monthly team meetings and participate in team-building efforts.
• Contribute to OP technical discussions, peer reviews, etc.
• Contribute content and collaborate via the OP-Wiki/Knowledge Base.
• Provide status reports to OP Account Management as requested.
  
  

About Us

At OP, we help you harness the power of technology for maximum impact. A technology consulting and solutions company, we offer advisory and managed services, innovative platforms, and staffing solutions across a wide range of fields, including AI, cybersecurity, enterprise architecture, and beyond. For nearly two decades, we’ve been challenging the status quo of the consulting industry, serving up fresh, ingenious thinking through a radically lean structure. Together, this strategy delivers unprecedented performance at an unparalleled pace for faster results that propel your business forward.