What are the responsibilities and job description for the Azure Databricks position at Prabhav Services Inc?
Responsibilities
Data Engineering: Design and build scalable ETL pipelines using Azure Databricks.
Data Migration: Migrate data from on-premises or legacy systems into Azure cloud environments.
Big Data Processing: Handle large datasets using Spark, Delta Lake, and Databricks notebooks.
Integration: Connect Databricks with Azure Data Lake, Synapse, Power BI, and other services.
Optimization: Tune Spark jobs for performance and cost efficiency.
Collaboration: Work with data scientists, analysts, and business stakeholders to deliver insights.
Required Skills
Azure Databricks: Strong hands-on experience with Databricks notebooks, clusters, and Delta Lake.
Apache Spark: Expertise in PySpark/Scala for distributed data processing.
Azure Ecosystem: Familiarity with Azure Data Lake, Synapse Analytics, Event Hub, and Power BI.
Programming: Proficiency in Python, Scala, or SQL.
Cloud Architecture: Understanding of cloud-native design, CI/CD, and DevOps practices.
Data Governance: Knowledge of security, compliance, and role-based access in Azure.
Ideal Candidate
Experience in end-to-end data migration and transformation projects.
Strong problem-solving and debugging skills in distributed environments.
Ability to collaborate across technical and business teams.
Excellent communication and documentation skills.