What are the responsibilities and job description for the Python Big Data Engineer position at Long Finch Technologies?
Job Description
We are seeking an experienced Python Big Data Engineer with strong expertise in Databricks, Python, and AWS to design, develop, and maintain scalable data engineering solutions. The ideal candidate will have hands-on experience building cloud-native data pipelines, implementing ETL processes, and developing robust data platforms in an AWS environment. Experience with infrastructure automation and modern application development is highly desirable.
Roles and Responsibilities
- Design, develop, and optimize scalable data pipelines using Databricks and Python.
- Build and maintain cloud-based data solutions on AWS.
- Develop and enhance ETL/ELT workflows for processing large-scale datasets.
- Implement data integration, transformation, and validation processes.
- Collaborate with cross-functional teams to understand business and technical requirements.
- Ensure data quality, reliability, security, and performance across platforms.
- Automate cloud infrastructure provisioning and deployment using Terraform.
- Support software engineering initiatives, including API and application development where required.
- Troubleshoot production issues and optimize existing data processing workflows.
Required Qualifications
- 8–10 years of overall IT experience with a strong focus on Data Engineering and Big Data technologies.
- Hands-on experience with Databricks, Python, and AWS services.
- Strong knowledge of ETL development, data modeling, and data pipeline architecture.
- Experience working with large-scale structured and unstructured datasets.
- Understanding of cloud-native architecture and distributed data processing.
- Experience with version control systems and CI/CD practices.
- Strong analytical, problem-solving, and communication skills.
Preferred Qualifications
- Experience with Node.js development.
- Hands-on experience with Terraform for Infrastructure as Code (IaC).
- Familiarity with Agile/Scrum development methodologies.
Required Skills
Must-Have Skills
- Databricks
- Python
- AWS Cloud Services
Preferred Skills
- Node.js
- Terraform