What are the responsibilities and job description for the Big Data Engineer position at Compsciprep LLC?
Role: Big Data Engineer
Location: Rockville, MD or Tysons, VA or Jericho, NY or New York, NY or Jersey City, NJ and Woodbridge, NJ
Duration: Long Term
Responsibilities:
- Design, develop, and maintain large-scale data processing pipelines using Big Data technologies (e.g., Hadoop, Spark, Python, Scala).
- Implement data ingestion, storage, transformation, and analysis of solutions that are scalable, efficient, and reliable.
- Stay current with industry trends and emerging Big Data technologies to continuously improve the data architecture
- Collaborate with cross-functional teams to understand business requirements and translate them into technical solutions.
- Optimize and enhance existing data pipelines for performance, scalability, and reliability.
- Develop automated testing frameworks and implement continuous testing for data quality assurance.
- Conduct unit, integration, and system testing to ensure the robustness and accuracy of data pipeline