What are the responsibilities and job description for the Databricks Engineer position at Robustware?
Job Details
Role: Databricks Engineer
Location: Irving, TX
10 years of experience in data engineering, with at least 3 years on Databricks.
- Strong proficiency in PySpark, SQL, and Delta Lake.
- Hands-on experience with Google Cloud Platform Dataproc.
Responsibilities:
1. **Administration**:
- Lead the installation and configuration of Databricks on Google Cloud Platform cloud platforms.
- Monitor platform health, performance, and cost optimization.
- Implement governance, logging, and auditing mechanisms.
2. **Development / Enhancements**:
- Design and develop scalable ETL/ELT pipelines using PySpark, SQL, and Delta Lake.
- Collaborate with data engineers and analysts to enhance data workflows and models.
- Optimize existing notebooks and jobs for performance and reliability.
3. **Operations, Support & Troubleshooting**:
- Provide L2/L3 support for Databricks-related issues and incidents.
- Troubleshoot cluster failures, job errors, and performance bottlenecks.
- Maintain technical documentation for platform setup, operations, and development standards