What are the responsibilities and job description for the Data Engineer position at 4i Americas?
Job Summary
We are looking for a Databricks Engineer with strong experience in building scalable data pipelines and migrating PySpark/Hive workloads from AWS EMR or legacy platforms to the Databricks Lakehouse. The ideal candidate should have expertise in Databricks, Delta Lake, Unity Catalog, and Spark technologies.
Key Responsibilities
- Design and develop scalable data pipelines using Databricks.
- Migrate PySpark, Hive, and legacy ETL workloads to Databricks Lakehouse.
- Build and optimize Spark SQL, PySpark, and Spark Declarative Pipelines (SDP).
- Implement Unity Catalog, data governance, and security best practices.
- Perform data validation, reconciliation, and quality assurance.
- Optimize performance, manage CI/CD pipelines, and support production deployments.
- Collaborate with architects, developers, and business teams.