What are the responsibilities and job description for the AWS Databricks Data Engineer position at Vertex Elite LLC?
Dear All,
Vertex Elite is currently seeking a qualified AWS Databricks Data Engineer to join our team.
If you or someone in your network might be a fit, please feel free to reach out for more details or share an updated resume.
Work Authorization: USC or Any valid US work authorization
Job Type: W2 Contract with Vertex Elite LLC
Location: Malvern, Pennsylvania
Required Qualifications:
- Design, develop, and maintain scalable data pipelines using Python, PySpark, Pandas, and advanced SQL.
- Develop data transformation and ingestion processes following object-oriented programming principles and best practices.
- Build and manage data solutions using Databricks Lakehouse, Workflows, and Delta Lake.
- Develop and optimize big data processing jobs using Apache Spark and, when required, Kafka.
- Implement data integration and ETL/ELT pipelines across various data sources and target systems.
- Use AWS services such as Amazon S3, AWS Glue, AWS Lambda, and Amazon Redshift to build cloud-based data solutions.
- Perform data cleansing, validation, transformation, and quality checks to ensure data accuracy and consistency.
- Write, optimize, and troubleshoot complex SQL queries for data extraction and transformation.
- Implement data partitioning, optimization, and performance-tuning techniques for large datasets.
- Collaborate with data architects, analysts, software engineers, and business stakeholders to understand data requirements.
- Develop and maintain infrastructure automation using Terraform.
- Containerize applications and data workloads using Docker, where applicable.
- Support CI/CD implementation and automated deployments using GitHub Actions.
- Monitor pipeline execution, troubleshoot failures, and resolve data processing issues.
- Follow security, governance, documentation, and coding standards across data engineering projects.
- Participate in code reviews, technical discussions, testing, and production support activities.
With Best Regards,
Manoj Visanagiri | Vertex Elite LLC | E-Verified Company | www.vertexelites.com