What are the responsibilities and job description for the Data Quality Engineer position at BuzzClan?
Role Context:
Design and run the data quality layer of our new Databricks lakehouse using Databricks Labs DQX (Data Quality eXtended). You will turn business rules from the legacy SQL platform into automated checks, quarantine bad records before they reach Gold tables, and give the business full visibility into data health.
Key Responsibilities:
* Profile source and Bronze data with the DQX profiler and turn output into reviewed quality rules.
* Define DQX checks (completeness, uniqueness, validity, ranges, referential, and custom SQL checks) in YAML/JSON config or code.
* Set criticality per rule (error vs. warn) and implement split/quarantine patterns for invalid rows.
* Embed DQX checks into Databricks Jobs and Lakeflow (DLT) pipelines for batch and streaming data.
* Store check definitions and results in Delta tables under Unity Catalog; build DQ dashboards and alerts.
* Translate legacy SQL validation logic (constraints, stored procedure checks) into equivalent DQX rules.
* Define DQ thresholds that act as quality gates for layer promotion (Bronze → Silver → Gold).
* Version-control rules and deploy them through CI/CD.
Key Deliverables & Outcomes:
* DQX checks live on every Silver and Gold table in scope.
* Invalid records quarantined with a documented triage process.
* A DQ dashboard that the business uses to track trends and SLA breaches.
Must-Have Skills:
* 3 years in data engineering or data quality roles.
* Hands-on PySpark and Spark SQL on Databricks.
* Practical experience with DQX, or strong experience with a comparable framework (Great Expectations, Soda, Deequ, DLT expectations) and willingness to ramp up on DQX quickly.
* Delta Lake and Unity Catalog fundamentals.
* Understanding of data quality dimensions and how to measure them.
* Git-based workflow and CI/CD basics.
Nice-to-Have Skills:
* Azure Data Factory orchestration of Databricks jobs.
* Databricks SQL dashboards, Lakehouse Monitoring, or Power BI for DQ reporting.
* Experience with SQL Server to cloud migrations.
* Databricks Certified Data Engineer Associate/Professional.