Databricks Governance Engineer

Position: Databricks Governance Engineer

Location: Atlanta, GA & Reston VA (Onsite)

Fulltime

NOTE: Candidate must be ready for an onsite final interview.

Job Description:

• The Senior Data Management Analyst supports enterprise data management and governance initiatives by identifying, profiling, cataloging, reconciling, and governing data across databases, cloud platforms, and file-based repositories.
• This role combines data analytics, technical expertise, cloud knowledge, and data governance. The analyst partners with data owners, stewards, engineers, developers, and governance teams to improve metadata quality, data discoverability, lineage, classification, data quality, and overall trust in enterprise data.
• The ideal candidate brings hands-on experience with Databricks, Collibra, Pentaho, SQL, AWS, and S3, along with the ability to translate technical data characteristics into meaningful business and governance metadata.

Key Responsibilities

• Identify, onboard, profile, and catalog enterprise data assets across databases, cloud platforms, data lakes, and AWS S3 repositories.
• Perform data discovery and profiling to assess structure, patterns, completeness, uniqueness, anomalies, and potential data quality issues.
• Perform detailed data reconciliation across multiple systems to validate completeness, accuracy, consistency, and integrity of data.
• Develop SQL-based reconciliation queries, reports, and analytical frameworks to compare data at the record, attribute, and aggregate levels.
• Analyze large datasets using SQL and data analysis tools to identify discrepancies, detect patterns, and determine root causes of data inconsistencies.
• Support metadata management, data classification, lineage, stewardship, ownership, and data quality processes.
• Develop and execute profiling and analytical logic using SQL, Databricks, Pentaho, Collibra, and related tools.
• Support automated scanning, metadata ingestion, classification, lineage, and cataloging within Collibra and integrated data platforms.
• Partner with data owners and stewards to maintain business definitions, technical metadata, classifications, and governance attributes.
• Analyze schemas, tables, files, pipelines, metadata, and reconciliation results to identify governance, lineage, and data quality gaps.
• Apply knowledge of AWS and cloud data architectures to support data discovery, profiling, cataloging, reconciliation, and governance.
• Serve as a liaison across governance, business, engineering, architecture, application, and platform teams.
• Translate governance and data quality requirements into technical requirements and collaborate with developers and engineers on implementation and issue resolution.
• Communicate data findings, discrepancies, risks, root causes, and recommendations clearly to technical and non-technical stakeholders.

Required Qualifications

• Bachelor’s degree in Computer Science, Information Systems, Data Analytics, Engineering, Data Management, or related field, or equivalent experience.
• Hands-on experience in data management, data governance, metadata management, data profiling, data reconciliation, data analysis, and data quality.
• Strong proficiency in SQL, data profiling, and reconciliation techniques.
• Experience analyzing large datasets and identifying data discrepancies, patterns, and root causes.
• Experience with Databricks or similar analytics/lakehouse platforms.
• Experience with Collibra or comparable enterprise data catalog and governance platforms.
• Working knowledge of AWS, particularly Amazon S3 and cloud-based data environments.
• Strong understanding of metadata, lineage, classification, business glossary, stewardship, ownership, and data quality concepts.
• Ability to work effectively with data engineers, developers, architects, data owners, stewards, and business stakeholders.
• Strong analytical, problem-solving, documentation, and communication skills.

Preferred Qualifications

• Experience with Pentaho; AWS Glue, Athena, Redshift, or Lake Formation; Databricks Unity Catalog–Collibra integration; Python/PySpark; automated scanning and lineage; enterprise data lakes; sensitive-data classification; and large-scale data reconciliation or validation frameworks is preferred.
• Ideal Candidate
• A technically proficient, collaborative data professional who operates at the intersection of Data Analytics Cloud Data Data Governance—comfortable investigating data, metadata, S3 files, schemas, pipelines, profiling and reconciliation results while applying a strong governance lens to make enterprise data accurate, trusted, traceable, documented, and discoverable.

About Smart IT Frame:

At Smart IT Frame, we connect top talent with leading organizations across the USA. With over a decade of staffing excellence, we specialize in IT, healthcare, and professional roles, empowering both clients and candidates to grow together

📩 Apply today or share profiles at dhamodharan@smartitframe.com