What are the responsibilities and job description for the Data Engineer position at Rivago Infotech Inc?
Experienced Senior Data Engineer (12 Years) to support large scale data platform modernization initiatives within a regulated banking environment.
The role focuses on designing and building enterprise-grade in-house frameworks, supporting high-volume batch and CDC-based incremental processing using Cloudera platform, and enabling ongoing Google Cloud Platform (GCP) modernization efforts.
Technology & Skill Requirements
• Apache Spark (PySpark and/or Scala) in large-scale production environments
• Cloudera Hadoop ecosystem (HDFS, Hive, YARN, Spark on Cloudera)
• Strong SQL expertise with complex transformations, performance tuning, and reconciliation logic
• Enterprise RDBMS experience with Oracle and MS SQL Server
• Batch ingestion, incremental ingestion, and CDC processing patterns
• CDC concepts and tooling (tool-agnostic: GoldenGate, Debezium, or equivalent)
• Data merge, deduplication, watermarking, checkpointing, and SCD handling
• Google Cloud Platform services including Dataproc , Composer and Dataplex
• Hybrid on prem to cloud data architecture and migration patterns
• Metadata-driven framework development and data quality validation techniques
• CI/CD pipeline implementation using enterprise tooling (GitHub Actions, Jenkins, DevOps)
• Git-based development workflows, code reviews, and automated testing practices
• Experience using Copilot or similar AI-assisted development tools safely and effectively in enterprise environments
• Logging, monitoring, alerting, and operational readiness practices
• Secure coding, access control, and compliance-aware development
• Documentation of design artifacts, runbooks, and operational procedures