What are the responsibilities and job description for the Databricks Data Engineer position at InnoCore Solutions, Inc.?
Responsibilities:
- Partner with business stakeholders and Global Product teams to understand data domains, business processes, and analytical requirements.
- Design, develop, and maintain scalable data pipelines and transformation jobs on the Databricks Lakehouse Platform to generate business metrics, KPIs, and curated datasets.
- Develop and optimize batch and streaming data pipelines using Databricks, Apache Spark (PySpark/Scala), and Delta Lake technologies.
- Build and maintain data ingestion frameworks from various enterprise source systems into AWS-based data lakes and Databricks environments.
- Implement and maintain Medallion Architecture (Bronze, Silver, and Gold layers) to support analytics and self-service reporting.
- Develop, optimize, and maintain Delta tables and data models to ensure high-performance data processing and analytics.
- Monitor, troubleshoot, and support Databricks workflows, clusters, and production data pipelines to ensure data reliability and availability.
- Implement data quality checks and validation frameworks to ensure data accuracy, completeness, timeliness, and consistency.
- Implement data governance and access controls using Unity Catalog and enterprise security standards.
- Optimize Spark jobs, cluster utilization, and query performance to improve scalability and cost efficiency.
- Build and deploy Databricks code, workflows, and infrastructure using GitHub Actions-based CI/CD pipelines.
- Implement monitoring, logging, alerting, and observability solutions using Datadog for proactive issue detection and operational support.
Requirements:
- 2 years of experience in Data Engineering and building enterprise-scale data platforms.
- 2 years of hands-on experience with Databricks and Apache Spark development using PySpark or Scala.
- Strong experience designing and implementing data pipelines on AWS and Databricks platforms.
- Experience with AWS services such as Amazon S3, AWS Glue, Amazon EC2, and IAM.
- Strong knowledge of Delta Lake, Unity Catalog, and Databricks Lakehouse architecture.
- Strong SQL skills and experience with data modeling and dimensional modeling techniques.
- Experience developing and deploying solutions using GitHub Actions and Git-based source control practices.
- Experience implementing monitoring and observability solutions using Datadog.
- Experience with data quality, metadata management, and data governance frameworks.
- Strong troubleshooting, performance tuning, and production support experience in cloud-based data platforms.
- Excellent communication and collaboration skills with the ability to work effectively with global teams.
- Databricks certifications such as Databricks Certified Data Engineer Associate or Professional.
Education:
- Bachelor's degree in Computer Science/Engineering or a related field