What are the responsibilities and job description for the Databricks Genie Engineer position at ConnectedX, Inc.?
Need consultants local to Plano, TX to attend Onsite interview.
Job Title: Databricks Genie Engineer
Location: Plano, TX Onsite
Duration: Long Term
Job Overview:
Need experienced Databricks Genie Engineer with data engineering experience and deep expertise in the Databricks Lakehouse platform. The candidate will design, build, and optimize scalable data pipelines, implement AI-powered data interactions using Databricks Genie, and drive enterprise-grade data solutions across cloud environments.
Key Responsibilities
Databricks & Lakehouse Engineering
- Design and implement scalable data pipelines using Databricks (PySpark, SQL).
- Architect and manage Delta Lake tables (Bronze, Silver, Gold layers).
- Optimize Spark jobs for performance, scalability, and cost efficiency.
- Implement Unity Catalog for data governance and access control.
- Manage cluster configuration, autoscaling, and job orchestration.
Databricks Genie Implementation
- Configure and optimize Databricks Genie for natural language-to-SQL analytics.
- Build semantic models and curated datasets to support Genie use cases.
- Improve AI-driven query accuracy and optimize prompt engineering.
- Collaborate with business users to enable self-service analytics.
- Troubleshoot and tune Genie-generated SQL for performance and correctness.
Data Architecture & ELT
- Design modern ELT architectures leveraging Delta Lake.
- Implement incremental processing (CDC, streaming with Structured Streaming).
- Develop data quality frameworks and validation pipelines.
- Handle schema evolution and medallion architecture best practices.
Cloud & Integration
- Work across AWS / Azure / Google Cloud Platform cloud platforms.
- Integrate with S3 / ADLS / GCS object storage.
- Implement CI/CD pipelines for Databricks deployments.
- Manage secrets, tokens, and secure connectivity.
- Integrate BI tools (Power BI, Tableau) and AI/ML workflows.
Required Qualifications
- 10 years of experience in Data Engineering.
- 5 years of hands-on experience with Databricks.
- Strong expertise in PySpark and Spark SQL.
- Deep understanding of Delta Lake and Lakehouse architecture.
- Experience implementing Databricks Genie or AI-driven analytics solutions.
- Strong experience with cloud platforms (AWS/Azure/Google Cloud Platform).
- Expertise in building scalable ELT pipelines.
- Solid understanding of data governance and security frameworks.
- Experience with CI/CD (Azure DevOps, GitHub Actions, Terraform, etc.).
Strong troubleshooting and debugging skills.