What are the responsibilities and job description for the AWS Data Engineer position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, Rivago infotech inc, is seeking the following. Apply via Dice today!
Role : AWS Data Engineer
Location : Mountain View CA (100% onsite)
Persistent system
Required Qualifications & Skills
Role : AWS Data Engineer
Location : Mountain View CA (100% onsite)
Persistent system
Required Qualifications & Skills
- 5 years of professional experience in software development, with 3 years focused on Big Data engineering.
- Expert proficiency in Python and its data stack (pandas, numpy).
- Deep, hands-on experience with Apache Spark and PySpark for large-scale data processing.
- Strong understanding of distributed computing principles and experience troubleshooting cluster performance issues using the Spark UI.
- Proficiency in SQL and experience with dimensional modeling and data warehousing concepts.
- Experience with AWS and its relevant big data services
- Pipeline Development: Design and implement robust, scalable, and fault-tolerant ETL (Extract, Transform, Load) processes and data pipelines using Python and PySpark on a distributed computing platform (e.g., Databricks, AWS EMR, or similar).
- Performance Optimization: Profile, tune, and optimize complex Spark queries and jobs to enhance processing speed and minimize resource utilization, including effective partitioning and caching strategies.
- Data Quality & Integrity: Implement data quality checks, validation rules, and reconciliation logic to ensure high standards of data integrity across all pipelines.
- Collaboration: Work closely with Data Scientists, Analysts, and other Data Engineers to understand data requirements and translate them into technical solutions.
- Code Management: Write clean, well-documented, and modular code, participate in code reviews, and maintain CI/CD pipelines for data solutions.
- Data Storage: Integrate Spark applications with various data sources and sinks, such as S3/HDFS, relational databases (SQL), and NoSQL stores.