What are the responsibilities and job description for the Senior Databricks Data Engineer position at Maddisoft?
job Title: Senior Databricks Data Engineer
Location: Austin, TX (Hybrid)
Interview Mode: In-Person
Job Summary
We are seeking an experienced Senior Databricks Data Engineer with 8 years of IT experience in designing, developing, and delivering enterprise data solutions. The ideal candidate will have extensive expertise in Databricks, Apache Spark, Python/Scala, SQL, and modern data lakehouse architectures. This role requires a highly motivated professional who can independently analyze business requirements, design scalable data solutions, optimize data pipelines, and deliver analytics-ready datasets that support business decision-making.
The candidate should be capable of planning and executing complex technical tasks with minimal supervision while collaborating effectively with business and technical stakeholders.
Key Responsibilities
- Analyze business requirements and translate them into scalable data engineering solutions.
- Design, develop, and optimize ETL/ELT pipelines using Databricks and Apache Spark (PySpark or Scala).
- Build and maintain Lakehouse architectures using Delta Lake and Medallion Architecture (Bronze, Silver, Gold).
- Develop and manage Lakeflow Declarative Pipelines (formerly Delta Live Tables) and orchestrate workflows using Lakeflow Jobs or similar scheduling tools.
- Design and implement dimensional data models, including Star and Snowflake schemas, to support enterprise reporting and analytics.
- Create Databricks SQL dashboards, Databricks Apps, and other analytical solutions that provide actionable business insights.
- Implement data governance, security, validation, and data quality frameworks to ensure reliable and trusted data.
- Optimize workload performance, scalability, monitoring, and operational efficiency while managing infrastructure costs.
- Perform system analysis, evaluate alternative technical solutions, and conduct cost-benefit analyses to recommend optimal approaches.
- Collaborate with cross-functional teams to gather requirements, document technical specifications, and deliver high-quality data solutions.
- Support CI/CD implementation for data engineering projects using Git-based workflows and DevOps best practices.
- Communicate technical concepts and project updates effectively to both technical and non-technical stakeholders.
Required Qualifications
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field (or equivalent experience).
- 8 years of experience designing, developing, deploying, or supporting enterprise technology solutions.
- 8 years of hands-on experience with Databricks and Apache Spark for building and optimizing ETL/ELT pipelines.
- Strong expertise in SQL and Python (or Scala) for large-scale distributed data processing.
- Experience with Delta Lake, Lakehouse architecture, and Medallion Architecture (Bronze, Silver, Gold).
- Experience implementing Lakeflow Declarative Pipelines (DLT) and scheduling jobs using Lakeflow Jobs or similar orchestration platforms.
- Strong experience in data warehousing concepts and dimensional data modeling (Star/Snowflake schemas).
- Experience developing dashboards and applications within Databricks using Databricks SQL and Databricks Apps.
- Strong understanding of data governance, data quality, data security, and enterprise data management best practices.
- Experience with CI/CD pipelines, Git, and DevOps practices for data engineering.
- Excellent analytical, problem-solving, verbal, and written communication skills.
- Ability to work independently while managing multiple complex initiatives.
Preferred Qualifications
- Experience working in public sector or government environments.
- Databricks Certified Data Engineer Associate or Professional certification.
- Experience with workflow orchestration tools such as Apache Airflow or similar platforms.
- Familiarity with cloud-based data platforms and modern data engineering best practices.