What are the responsibilities and job description for the Senior Python/PySpark/Spark Developer position at Avenues International, Inc.?
Job Details
Title: Senior Python/PySpark/Spark Developer
Location: Princeton areas NJ (Three days onsite is must)
Duration: One Year
Project Description:
Migration of legacy data ingestion pipelines to a shared data management system (BDMS). The work will involve:
Datasets will be loaded into BDMS
New tools will be created to allow data analysts to modify the datasets
Analysts will validate and cleanse the data which will then be delivered to AWS S3
Data will then be enriched using technologies like Databricks, AWS Lambda & Step Functions and Jupyter Notebooks/Workflows and loaded in Delta tables.
Must have skills:
4 years of experience programming in Python or Java
Solid understanding of database systems and SQL including Oracle and PostgreSQL
A Degree in Computer Science, Engineering, Mathematics, similar field of study or equivalent work experience.
AWS experience including services like S3, Lambda, CloudWatch
Highly Desirable skills:
Experience with data migrations from legacy to cloud.
Knowledge of Spark, Databricks and IaC(Terraform/CloudFormation)
Santy Kumar
Senior Business Manager
Avenues International Inc.
Email:
Salary : $80 - $90
