What are the responsibilities and job description for the KAFKA, REDIS AND HADOOP DEVELOPER position at VeriiPro?
We are seeking an experienced Kafka, Redis, and Hadoop Developer with strong expertise in big data technologies, real-time data processing, and distributed systems. The ideal candidate will have hands-on experience with Apache Kafka, Redis, Hadoop, PySpark, and Python to design, develop, and maintain scalable, high-performance data processing solutions.
Key Responsibilities
Key Responsibilities
- Design, develop, and maintain real-time data streaming applications using Apache Kafka, including KStreams and KTable.
- Develop scalable big data processing solutions using Hadoop and PySpark.
- Implement and optimize data storage and retrieval using NoSQL databases such as Redis, MongoDB, and HBase.
- Develop data processing and integration pipelines using Python, Scala, Java, and Bash/Shell scripting.
- Build reliable distributed systems with efficient data partitioning, fault tolerance, and high availability.
- Work with streaming technologies such as Apache Flink and Apache Storm.
- Leverage cloud platforms such as AWS, Azure, or GCP for big data processing and deployment.
- Apply data security and integration best practices, including Kerberos authentication.
- Perform data analysis and support predictive modeling using statistical techniques and machine learning algorithms.
- Develop and maintain visual analytics and reporting solutions using Tableau.
- Troubleshoot performance issues, optimize data pipelines, and ensure data quality and reliability.
- Collaborate with cross-functional teams to deliver business-critical data solutions.
- Bachelor's or Master's degree in Science, Engineering, Computer Science, or a related field.
- 9–12 years of overall professional experience in software development and data engineering.
- Strong hands-on experience with Apache Kafka, Redis, Hadoop, and PySpark.
- Proficiency in Python and familiarity with Scala, Java, and Bash/Shell scripting.
- Strong understanding of real-time streaming, distributed computing, and big data architecture.
- Experience with NoSQL databases and cloud-based data processing platforms.
- Knowledge of data security, data integration, and visualization tools such as Tableau.
- Excellent problem-solving, analytical, and communication skills.