What are the responsibilities and job description for the Data Engineer position at Saransh Inc?
Brighton, MA (Onsite)
Long Term
Key Responsibilities
Long Term
Key Responsibilities
- Design, develop, and implement end-to-end big data solutions across the enterprise.
- Develop and maintain big data applications using Apache Spark, Scala, AWS Glue, AWS Lambda, SNS/SQS, and CloudWatch.
- Build and optimize data pipelines, ensuring high performance, scalability, and reliability.
- Collaborate with cross-functional teams to design and document integration and application technical designs.
- Conduct peer reviews of functional and design documentation to maintain high-quality standards.
- Develop, configure, and perform unit testing and code reviews to ensure coding best practices.
- Troubleshoot and resolve complex issues during testing phases and identify root causes efficiently.
- Perform performance testing and optimize system performance.
- Manage and maintain SQL-based databases, preferably Amazon Redshift.
- Utilize Snowflake for data warehousing (experience in Snowflake is an added advantage).
- Implement ETL/ELT processes and ensure data quality across various systems.
- Work with Git repositories and manage CI/CD deployment pipelines for continuous integration and delivery.
- Provide production support, including troubleshooting and environment tuning.
- Ensure adherence to best practices and technical standards throughout the project lifecycle.
- 6 years of relevant experience in big data design and development.
- Proficiency in Scala and/or Python for application development.
- Strong expertise in Spark, AWS Glue, Lambda, SNS/SQS, and CloudWatch.
- Hands-on experience with ETL/ELT frameworks and data integration.
- Advanced knowledge of SQL, with experience in Redshift preferred.
- Familiarity with Snowflake and cloud-based data solutions is advantageous.
- Experience with CI/CD processes, Git, and production support in large-scale environments.