What are the responsibilities and job description for the Sr. Data engineer position at DATAECONOMY?
Role: Data engineer- Databricks
Location: Rutherford, NJ
Job description:
• Good decision-making and problem-solving skills
• Solid understanding of Databricks fundamentals/architecture and have hands on experience in setting up Databricks cluster, working in Databricks modules (Data Engineering, Machine Learning and SQL warehouse).
• Strong experience in coding custom ETL pipelines using spark and deploy in Databricks.
• Knowledge of medallion architecture, DLT and unity catalog within Databricks.
• Understanding of core AWS services, uses, and AWS architecture best practices
• Hands-on experience in different domains, like database architecture, business intelligence, machine learning, advanced analytics, big data, etc.
• Solid knowledge on Airflow
• Solid knowledge on CI/CD pipeline in AWS technologies
• Solid programming background on scala spark and/or pySpark
• Experience with Docker and Kubernetes is a plus