What are the responsibilities and job description for the Senior PySpark Developer / Engineer position at IT-SCIENT?
Role: Senior PySpark Developer / Engineer
Location: Dallas, TX (3 Days onsite/ week)
Full Time Position
Job Description:
Must-Have Technical Skills: PySpark, SQL and UNIX
PySpark Expertise-
Strong hands-on experience with PySpark architecture understanding
Clear understanding of how PySpark jobs execute in the background
Ability to read, write, analyze, and enhance PySpark scripts and configurations
SQL Proficiency:
Strong SQL skills
Experience working with SQL as part of data processing and PySpark job execution
Unix Shell Scripting:
Should have good understanding of Unix shell scripting.
Ability to understand backend Unix scripts supporting PySpark workflows
Experience in reading, analyzing, writing, and modifying shell scripts for PySpark integration
ETL Knowledge:
Hands-on experience with ETL processes using any programming language or framework
Ability to understand and work with complementary ETL workflows alongside PySpark
Job Scheduling & Automation
Experience with at least one enterprise scheduling/automation tool such as: Autosys/ Control‑M/ Airflow/ or similar tools used in production environments
Nice-to-Have Skills
Experience working with finance domain applications
Familiarity with related frameworks or tools that enhance PySpark-based workflows
''The information contained in this electronic message and any attachments to this message are intended for the exclusive use of the addressee(s) and may contain proprietary, confidential or privileged information. If you are not the intended recipient, you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately and destroy all copies of this message and any attachments. WARNING: Computer viruses can be transmitted via email. The recipient should check this email and any attachments for the presence of viruses. The company accepts no liability for any damage caused by any virus transmitted by this email. ''