What are the responsibilities and job description for the Big Data engineer position at VySystems?
Key Responsibilities:
• Data Pipeline Development: Designing, developing, and maintaining data pipelines to efficiently move and transform large datasets from various sources.
• System Design and Architecture: Creating the overall architecture of the big data platform and ensuring its scalability, performance, and reliability.
• Data Integration and Processing: Integrating data from diverse sources, ensuring data quality, and implementing data processing workflows.
• Performance Optimization: Monitoring system performance, identifying bottlenecks, and optimizing data processing for efficiency. Essential Skills:
• Programming Languages: Proficiency in languages like Java , Python, Scala,
• Big Data Technologies: Experience with frameworks like Hadoop, Spark, Kafka, Hive, and other related tools.
• Database Management: Knowledge of database design, data modeling, and various database technologies.
• Data Analysis and Visualization: Understanding of data analysis techniques and data visualization tools.
• Problem-Solving and Analytical Skills: Ability to analyze complex data problems and develop effective solutions.
• Communication and Collaboration: Effective communication and teamwork skills. Experience: 8 year experience for Big data developer and 12 year experience for Big data lead.