What are the responsibilities and job description for the Bigdata Engineer position at Digital Dhara LLC?
We are seeking a highly skilled Big Data Engineer to join our data engineering team for a leading banking client. The ideal candidate will have strong experience in building and managing large-scale data pipelines, implementing data processing frameworks, and working with banking or financial datasets.
Key Responsibilities
- Design, develop, and maintain scalable Big Data pipelinesand ETL workflows using tools such as Spark, Hadoop, Kafka, and Airflow.
- Collaborate with data scientists, analysts, and business stakeholders to deliver data solutionssupporting core banking, risk, fraud, and customer analytics.
- Implement data ingestion, cleansing, transformation, and storageprocesses from multiple banking systems (e.g., core banking, credit card, payments, CRM).
- Optimize data lake and data warehouse architecturesfor performance, reliability, and security.
- Manage and monitor real-time and batch data processingon cloud and on-prem platforms.
- Ensure data governance, lineage, and compliancewith regulatory frameworks (e.g., GDPR, RBI, Basel).
- Participate in code reviews, performance tuning, and production support activities.
Technical Skills Required
- Big Data Frameworks:Hadoop, Spark (PySpark/Scala/Java), Hive, HBase, Kafka, Flink
- Data Integration Tools:NiFi, Airflow, Talend, Informatica (optional)
- Programming Languages:Java, Scala, Python
- Databases:Oracle, Teradata, Snowflake, PostgreSQL, or any RDBMS
- Cloud Platforms:AWS (Glue, EMR, S3, Redshift), Azure (Data Lake, Synapse), or Google Cloud Platform (BigQuery)
- Version Control & CI/CD:Git, Jenkins, Docker, Kubernetes (nice-to-have)
- Security & Compliance:Understanding of banking data privacy, masking, and encryption standards