What are the responsibilities and job description for the Data Engineer position at Quadrant IQ Solutions LLC?
Job Title: Senior Data Engineer (Spark / AWS / Snowflake)
Location: New york, New York
Job Type: W2
Visa's Accepted: H4-EAD, GC-EAD, L2S, GC, USC, H1B
Must-Have Skills
- Python & PySpark – Expert-level proficiency with very strong Apache Spark experience (large-scale data processing, optimization, performance tuning).
- AWS Cloud
- ECS (Elastic Container Service) – Required
- AWS Glue and/or AWS Lake Formation – Strongly preferred
- Data Warehousing
- Snowflake or Teradata (hands-on experience with large analytical datasets)
Core Responsibilities
- Design, build, and optimize high-performance Spark-based data pipelines using Python/PySpark.
- Develop and maintain ETL/ELT workflows for large-scale structured and semi-structured data.
- Deploy and manage data workloads on AWS ECS.
- Implement data ingestion, transformation, and governance using AWS Glue / Lake Formation.
- Integrate data pipelines with Snowflake or Teradata for analytics and reporting.
- Ensure scalability, reliability, and performance of distributed data processing systems.
- Collaborate with analytics, platform, and cloud engineering teams.