What are the responsibilities and job description for the Databricks Data Engineer| Senior position at Jobgether?
This position is listed on behalf of a partner company, who manages all applications and next steps. Our partner is looking for a Databricks Data Engineer| Senior based in Brazil.
This role is designed for an experienced Data Engineer who will help modernize large-scale data platforms by migrating legacy data warehouse pipelines into a Databricks-based Lakehouse environment.
You will play a key role in rebuilding data workflows, ensuring data quality, and enabling scalable analytics solutions.
The position requires strong expertise in PySpark, Spark SQL, Python, and modern data engineering practices.
You will collaborate with engineering teams to transform complex business rules into reliable, production-ready pipelines.
Working across cloud technologies and advanced data platforms, you will contribute to a critical digital transformation initiative.
This is an opportunity to apply your technical expertise while improving performance, governance, and operational efficiency across enterprise data systems.
The role involves leading the reconstruction, optimization, and validation of data pipelines while ensuring alignment with modern Lakehouse architecture standards. Key responsibilities include:
- Rebuild legacy data warehouse pipelines using Databricks, PySpark, and Spark SQL based on specifications created through reverse engineering.
- Implement bronze, silver, and gold data layers following the Medallion architecture, ingestion standards, and project reconstruction guidelines.
- Execute migration waves by business domain while maintaining coexistence between legacy systems and the new platform until final cutover.
- Develop business logic transformations and implement automated testing across data pipelines.
- Perform data reconciliation and validate parity between legacy data warehouse outputs and new Lakehouse implementations.
- Optimize pipeline performance and cloud costs through techniques such as partitioning, OPTIMIZE/Z-ORDER strategies, and job sizing improvements.
- Contribute to technical documentation, migrated business rules, and prioritization of future migration activities.
- Collaborate with engineering teams to improve data quality, reliability, and delivery processes.
Requirements
The ideal candidate is an experienced Data Engineer with strong knowledge of scalable data platforms, cloud environments, and data migration initiatives. Required qualifications include:
- 5 years of experience in Data Engineering roles.
- Advanced proficiency in PySpark and Python, including large-scale batch pipeline development, testing practices, and engineering standards.
- Strong SQL expertise and experience with complex transformations, performance tuning, and relational or dimensional data modeling.
- Hands-on experience with Databricks and Delta Lake, including production workflows, jobs, Medallion architecture, Delta Live Tables, Lakeflow, and Asset Bundles.
- Experience migrating or rebuilding ETL pipelines by translating business rules from legacy systems into Spark-based solutions with CDC and batch ingestion patterns.
- Knowledge of AWS data services such as S3, Glue, EMR, Athena, Lambda, DMS, and Step Functions.
- Experience with Azure Synapse environments, including SQL pools and pipelines, to support reverse engineering activities.
- Strong understanding of data quality practices, automated testing, data quality gates, and legacy versus new platform reconciliation.
- Experience using Git and CI/CD practices for data pipeline development.
- Ability to communicate effectively, collaborate with technical teams, and work independently in complex transformation projects.
Preferred qualifications:
- Experience analyzing DataStage jobs for reverse engineering purposes.
- Previous involvement in Data Warehouse migration programs toward Lakehouse architectures.
- Knowledge of Unity Catalog, permissions management, data lineage, and data contracts.
- Experience in financial services or credit-related environments.
- Databricks certifications such as Data Engineer Associate or Professional.
Benefits
- Opportunity to work on a large-scale data modernization and Lakehouse transformation initiative.
- Remote work flexibility within Brazil.
- Exposure to advanced technologies including Databricks, Delta Lake, Spark, AWS, and Azure platforms.
- Opportunity to collaborate with experienced engineering teams on high-impact data projects.
- Professional growth through access to emerging AI, cloud, and data engineering practices.
- Participation in a technology-driven environment focused on innovation and continuous learning.
How Jobgether works:
We use an AI-powered matching process to ensure your application is reviewed quickly, objectively, and fairly against the role's core requirements. Our system identifies the top-fitting candidates, and this shortlist is then shared directly with the hiring company. The final decision and next steps (interviews, assessments) are managed by their internal team.
We appreciate your interest and wish you the best!
Why Apply Through Jobgether?
Data Privacy Notice: By submitting your application, you acknowledge that Jobgether will process your personal data to evaluate your candidacy and share relevant information with the hiring employer. This processing is based on legitimate interest and pre-contractual measures under applicable data protection laws (including GDPR). You may exercise your rights (access, rectification, erasure, objection) at any time.
#LI-CL1