What are the responsibilities and job description for the Databricks Admin/Unity Catalog position at Millennium Software, Inc.?
Role Overview:We are seeking a highly skilled Databricks Administrator to manage our data ecosystem. In this role, you will be responsible for provisioning Databricks and Unity Catalog services, maintaining organizational governance, and automating data-related processes. You will collaborate with cross-functional teams to translate business requirements into scalable, secure data solutions.
Key Responsibilities:
- Provisioning & Architecture: Manage distributed workspaces, workspace policies, and cluster sizing/capacity. Implement data architecture leveraging Databricks, Unity Catalog, Privacera, and Collibra.
- Governance & Compliance: Ensure data security through rigorous user/group management, identity federation, and role-based access control (RBAC).
- Pipeline Optimization: Troubleshoot ETL bottlenecks and ML workloads. Develop UDFs (SQL/Python) and create data masking processes.
- Collaboration: Work alongside data scientists and analysts to operationalize ML models in Batch and Real-Time pipelines.
Required Skills & Experience:
- Core Databricks: Account/Workspace administration, configuration across environments, and compute resource optimization.
- Unity Catalog: Deep expertise in metastores, catalogs, schemas, external locations, and Delta sharing.
- Technical Proficiency: Strong scripting experience in Python and SQL.
- Security & Compliance: Proven experience with security compliance procedures and implementing guardrails for data sharing.
- Engineering Mindset: Ability to guide teams on pipeline building and performance optimization.
Preferred Qualifications:
- Understanding of Agentic Architecture.
- Familiarity with the data requirements of common ML/AI use cases.
- Experience navigating multi-step enterprise approval processes (Architecture, Security, Governance).
Salary : $75 - $80