What are the responsibilities and job description for the Data Engineer position at Holistic Partners, Inc?
Job Description
Role:: Data Engineer - Databricks
Location:: 02171, Quincy, Massachusetts, United States
Duration:: Full Time
Experience:: 6 to 12 years
Must Have Technical/Functional Skills
Must have at least 6 years of extensive Data Engineering Experience in Databricks, ETL/ELT using Data pipelines, SQL/Procedure experience.
• Must have 8 Years of hands-on development proficiency in implementing Databricks solutions using Scala and Spark with Data frames and Notebooks/SQL
• Must have 6 Years of hands-on development and performance tuning/enhancement experience in Scala/PySpark and SQL.
• Must have 6 Years of hands-on development experience in working on Azure Cloud.
• Must have strong experience with Databricks, including developing and optimizing Spark jobs, data transformations, and data processing workflows.
• Must have experience and a good understanding on the below topics
• Databricks Delta Lake storage
• Unity Catalog
• Autoloader
• Lakehouse Architecture
• Medallion Architecture
• Databricks Serverless Options.
• Dataset and Data frame
• Databricks Clusters Configuration and Sizing
• Databricks Job Orchestration
• Spark Structured Streaming
• Must have experience with complex and large volume data streaming and data transformations using Azure Databricks.
• Must have extensive experience in Datawarehouse/Data Lakehouse implementations.
• Must have experience with monitoring, debugging, and resolving issues in live Databricks jobs.
• Must have strong hands-on expertise in troubleshooting DevOps pipelines, Azure, and AKS services
• The candidate should be able to provide Data Ingestion, Streaming and Transformations solutions and do development work.