What are the responsibilities and job description for the Microsoft Databricks Engineer position at 4 Consulting Inc?
Job Title: Microsoft Databricks Engineer
Location: Fairfield, OH
Duration: long term
Job ID: 4CI -7873
Required Qualifications:
- Hands-on experience with Databricks and data engineering technologies.
- Strong proficiency in SQL, Python, and Apache Spark.
- Experience building data pipelines and ETL/ELT solutions.
- Experience with cloud platforms, preferably Microsoft Azure.
- Knowledge of data modeling, data architecture, and database design.
- Experience with relational and NoSQL databases.
- Understanding of data governance, data quality, and security best practices.
- Experience with workflow orchestration tools such as Airflow or Luigi.
- Familiarity with Medallion Architecture and Data Mesh concepts.
- Design, develop, and maintain scalable data pipelines and ETL/ELT processes.
- Build and optimize data lakehouse solutions using Databricks and cloud technologies.
- Collaborate with business stakeholders, data analysts, and engineering teams to gather requirements and deliver reliable data products.
- Implement and support data architectures following Medallion (Bronze, Silver, Gold) patterns and modern data management principles.
- Develop and optimize data processing workflows using SQL, Python, and Apache Spark.
- Ensure data quality, governance, security, and compliance across the data lifecycle.
- Monitor, troubleshoot, and enhance the performance of data platforms and pipelines.