What are the responsibilities and job description for the Data Architect (AI/ML & Databricks) position at Diligent Tec, Inc?
Job Title: Data Architect (AI/ML & Databricks)
Location: Denver, CO (Onsite)
Job Type: Full Time - (Salary Benefits)
Min Experience- 14 Years
Required Skills / Qualifications
14 years of experience in data engineering or data architecture, with a strong focus on Databricks (4-5 years) and AI/ML enablement
Strong hands-on experience with Apache Spark and Delta Lake
Expertise in AI/ML pipeline integration using Databricks MLflow and/or custom model deployment strategies
Strong knowledge of Apache Airflow, Databricks Jobs, and cloud-native orchestration patterns
Experience with Structured Streaming, Kafka, and real-time analytics frameworks
Proven ability to design and implement cloud-native data architectures
Solid understanding of data modeling, Lakehouse design principles, and data lineage/tracking using Unity Catalog
Excellent communication skills and experience working with stakeholders
Preferred Qualifications
Databricks Data Engineering Professional Certification (highly desirable)
Experience transitioning from in-house data platforms to Databricks / cloud-native environments
Hands-on experience with Delta Lake, Unity Catalog, and performance tuning in Databricks
Expertise in Airflow DAG design, dynamic workflows, and production troubleshooting
Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices
Exposure to AI/ML model integration in real-time and batch data pipelines
Exposure to MLOps, MLflow, Feature Store, and production model monitoring
Experience with LLM/GenAI enablement, vectorized data, embedding storage, and Databricks integration (added advantage)
Location: Denver, CO (Onsite)
Job Type: Full Time - (Salary Benefits)
Min Experience- 14 Years
Required Skills / Qualifications
14 years of experience in data engineering or data architecture, with a strong focus on Databricks (4-5 years) and AI/ML enablement
Strong hands-on experience with Apache Spark and Delta Lake
Expertise in AI/ML pipeline integration using Databricks MLflow and/or custom model deployment strategies
Strong knowledge of Apache Airflow, Databricks Jobs, and cloud-native orchestration patterns
Experience with Structured Streaming, Kafka, and real-time analytics frameworks
Proven ability to design and implement cloud-native data architectures
Solid understanding of data modeling, Lakehouse design principles, and data lineage/tracking using Unity Catalog
Excellent communication skills and experience working with stakeholders
Preferred Qualifications
Databricks Data Engineering Professional Certification (highly desirable)
Experience transitioning from in-house data platforms to Databricks / cloud-native environments
Hands-on experience with Delta Lake, Unity Catalog, and performance tuning in Databricks
Expertise in Airflow DAG design, dynamic workflows, and production troubleshooting
Experience with CI/CD pipelines, Infrastructure-as-Code (Terraform, ARM templates), and DevOps practices
Exposure to AI/ML model integration in real-time and batch data pipelines
Exposure to MLOps, MLflow, Feature Store, and production model monitoring
Experience with LLM/GenAI enablement, vectorized data, embedding storage, and Databricks integration (added advantage)