What are the responsibilities and job description for the Databricks Data Engineer- azure databricks, Scala is Must position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, Quadrant Technologies, is seeking the following. Apply via Dice today!
Databricks Data Engineer- azure databricks, Scala is Must
Location : Deerfield, IL (Onsite)
Full Time Only
Job Description
Quadrant Technologies 1- (Cell)
Databricks Data Engineer- azure databricks, Scala is Must
Location : Deerfield, IL (Onsite)
Full Time Only
Job Description
- Mandatory skills: Databricks, Spark, Azure, Python, Cosmos DB, Azure DevOps (GitHub, CI/CD pipelines, Boards etc.), Docker & Azure Kubernetes Service, Graffana, JUnit, Postman, SonarQube
- Additional Skills: 3 6 years of experience in data engineering on cloud data platforms. Handson experience building Spark jobs with Scala/Spark and/or PySpark on Databricks. Experience ingesting data from batch and streaming sources into ADLS Gen2 using Delta or Apache Iceberg tables. Good SQL skills for joins, aggregations, and data quality checks. Understanding of core Azure data services (Event Hubs/Kafka, Data Factory/Databricks Workflows, Key Vault). Experience working with Gitbased workflows and CI/CD in Azure DevOps or GitHub.
- Good to have skills: Exposure to Spark Structured Streaming for near realtime use cases. Experience with data quality tools or frameworks and writing unit/integration tests for data pipelines. Familiarity with data modeling and performance considerations in Lakehouse environments.
- Responsible for building data products in Databricks using Scala/Spark
- Responsible for Ops work managing production for the data products developed and deployed to production
- Responsible for testing data products based on product specification and end to end validation
- Set up monitoring, logging, and alerting Spark jobs and data pipelines using Azure Monitor/Log Analytics or similar tools
- Coordinate with offshore team in India
Quadrant Technologies 1- (Cell)