What are the responsibilities and job description for the Big data Pyspark Developer position at Quantum World Technologies Inc.?
Job Title: Big data Pyspark Developer
Work Location : Irving, Texas/ Tampa, FL
Job Summary
- Exp 6 Years Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
- Unix SQL and Shell Scripting experience is a must have
- Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines
- Strong commandline knowledge in UnixLinux with Shell scripting using Bash Kornshell or Perl and File processing using awk scripts
- Expertise in SQL querying and complex joins
- Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
- Expertise with complex Data workflows with Apache AirFlow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
- Strong Analytical skills and expertise on SparkSQL for Data analysis and validation ensuring the delivery of clean queryready datasets for business consumption
- Expertise in Data quality checks and monitoring
- Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
- Handson with Automation Framework Design for ETL and API
- SME in Data Analysis Database testing Messaging queues
- Experience with coding standards code reviews source management build processes CICD pipeline
Skills
Mandatory Skills : Apache Spark, Big Data Hadoop Ecosystem, Python, Python for DATA, SparkSQL