What are the responsibilities and job description for the Hadoop Data Engineer position at QODE?
responsible for designing, developing, and maintaining large-scale data processing systems within a distributed Hadoop ecosystem. The role focuses on enabling data-driven decision-making across banking operations, risk management, compliance, and customer analytics.
Key Responsibilities
Technical Skills
Key Responsibilities
- Design, develop, and maintain scalable data pipelines using Hadoop ecosystem tools (HDFS, Hive, Spark, Sqoop, Kafka).
- Build and optimize ETL/ELT processes to support data ingestion from multiple banking systems.
- Develop and manage big data solutions for structured and unstructured data.
- Collaborate with data analysts, data scientists, and business stakeholders to deliver data solutions.
- Ensure data quality, integrity, and governance aligned with banking and regulatory standards.
- Perform performance tuning and optimization of Hadoop/Spark jobs.
- Implement data security controls to comply with financial regulations (e.g., PCI, SOX).
- Support real-time and batch data processing frameworks.
- Troubleshoot production issues and provide continuous support for data platforms.
- Work with cloud platforms (e.g., AWS, Azure) for modern data solutions.
Technical Skills
- Strong experience with:
- Hadoop ecosystem (HDFS, MapReduce, Hive, HBase)
- Apache Spark (Scala/Python)
- SQL & NoSQL databases
- ETL tools (Informatica, Talend, or similar)
- Kafka or other streaming tools
- Proficiency in programming:
- Python / Java / Scala
- Experience with:
- Data warehousing concepts
- Workflow orchestration tools (Airflow, Oozie)
- Unix/Linux environments
- Knowledge of cloud data platforms (AWS EMR, Azure Data Lake) is a plus
- Understanding of banking and financial services data
- Exposure to risk, compliance, or fraud analytics is preferred
- Strong problem-solving and analytical abilities
- Excellent communication and collaboration skills
- Ability to work in Agile/Scrum environments
- Bachelor’s or Master’s degree in:
- Computer Science, Information Technology, or related field
- Typically 5–10 years of experience in data engineering or big data development