What are the responsibilities and job description for the Databricks / Python Data Engineer position at TekDallas?
Databricks / Python Data Engineer
📍 Location: Ohio(Hybrid)
💼 Duration: Long Term Contract
We are seeking an experienced Databricks / Python Data Engineer to design, develop, and optimize enterprise-scale data pipelines and Lakehouse solutions using Python, Databricks, Apache Spark and PySpark.
🔑 Key Responsibilities
- Build scalable ETL/ELT pipelines using Python, PySpark and Databricks
- Develop batch and near-real-time data pipelines
- Implement Bronze / Silver / Gold Medallion Architecture
- Work with Delta Lake and cloud object storage
- Optimize Spark jobs for performance, scalability and cost
- Implement data quality, schema validation, lineage and observability
- Develop enterprise integrations using APIs, files and cloud platforms
- Manage Databricks Workflows / DLT / Unity Catalog
- Build automated CI/CD pipelines
- Use GitHub-based development workflows
- Leverage GitHub Copilot, Claude, Cursor or similar AI coding assistants
- Participate in architecture, code reviews and Agile/Scrum ceremonies
- Mentor junior/mid-level data engineers
🛠️ Required Skills
- 5 years Data Engineering
- Strong Python
- Databricks
- Apache Spark / PySpark
- SQL
- ETL/ELT
- Delta Lake
- Lakehouse / Medallion Architecture
- Data modeling
- Data quality & schema evolution
- Spark performance tuning
- Cloud object storage
- CI/CD & Git
- Workflow orchestration
⭐ Preferred
- Delta Live Tables
- Databricks Workflows
- Unity Catalog
- Structured Streaming / Kafka
- dbt
- Terraform / IaC
- Azure Data Factory / Azure DevOps
- Data observability
- Enterprise API integrations
- Claude / GitHub Copilot / Cursor / AI-assisted development