What are the responsibilities and job description for the Databricks Data Operation Engineer position at Avance Consulting?
Job Title: Databricks Data Operation Engineer
Location: Philadelphia( PA
Duration: Full-Time
Required Skill and Experience
o Azure Data Factory (ADF) pipelines, triggers, integration runtime / SHIR dependencies (as applicable).
o Databricks jobs/workflows (cluster usage patterns, job failures, notebook execution issues, library/secret-related failures).
o ADLS data availability, folder/file arrival patterns, permissions/ACL-related access failures (in collaboration with platform team where required).
Preferred Skill and Experience
o Execute operational data validation and reconciliation checks (row counts, completeness, duplicates, timeliness, schema drift, threshold breaches).
o Log, track, and support resolution of DQ issues via defined processes; partner with DG/DQ teams (e.g., Collibra/Informatica) where rule ownership and stewardship workflows exist.
o Maintain “data trust” by ensuring incidents are not closed without confirming data correctness and downstream readiness (where applicable).