Job Description

POSITION GENERAL DUTIES AND TASKS :

Technical Lead with 12 yrs of experience as well as expertise in data engineering and analytics in In-depth knowledge of Apache Spark, Pyspark and Databricks architecture. Strong programming skills in languages such as Python or Scala. Experience with data engineering concepts, ETL processes, and data modeling. Proven experience as a Technical Lead with a focus on Databricks and big data technologies on Azure Cloud Platform. Design and implement scalable and efficient data processing architectures using Databricks. Handson experience in writing python code, pyspark notebooks and Azure Cloud, Configure Databricks clusters and ensure adherence to best practices and coding standards Make strategic decisions regarding data organization, storage, and processing. Lead and mentor a team of data engineers and data scientists. Provide technical guidance and support to ensure the team's success. Oversee the implementation of data processing workflows, analytics, and machine learning models using Databricks. Optimize the performance of big data processing tasks and analytics jobs. Collaborate with cross-functional teams, including data science, business intelligence, and infrastructure teams. Ensure seamless integration of data pipelines with various systems. Ensure consistency, quality, and maintainability of the codebase. Optimize queries and resolve technical challenges during development and production. Manage the technical aspects of projects, including timelines and resource allocation. Coordinate with stakeholders to ensure project goals are met. Excellent leadership and communication skillsBasic Qualifications:
8 years of experience in Databricks