What are the responsibilities and job description for the Azure Data Engineer position at Programmers.io?
Role Descriptions:
- Design and build batch and streaming data pipelines using Azure, Databricks, and Kafka.
- Develop ETL/ELT solutions using Spark, SQL and Python/Pyspark, or Scala
- Implement and support Data Lake / Lakehouse architectures
- Integrate data from legacy and modern platforms. Preferred postgres DB migration to Azure Cloud.
- Expertise in handling Data migration using API’s.
- Apply data quality checks and support data governance basics (metadata, lineage, access controls)
- Optimize Spark jobs for performance, reliability, and cost
- Collaborate with cross-functional teams, Product owners, Admin teams, BSAs, Scrum & Project managers
Required Skills & Experience:
- 8 years of experience in data engineering in azure using Azure Databricks.
- Strong experience with Azure Data Lake, Databricks, ADF, Postgres.
- Strong Data Engineering experience in Big Data (Cloudera)
- Hands-on expertise in SQL, Python and PySpark or Scala
- Proven experience with batch & real-time ingestion
- Experience with Kafka and big data processing
Added good to have experience with:
- Informatica PowerCenter
- Informatica IDMC or equivalent cloud data integration tools
- SQL Server, Oracle, Azure SQL
- Cosmos DB
- Unix/Linux
- Maestro (or similar enterprise schedulers)
- Data tools such as Precisely or equivalent
Added Advantage
- Insurance domain knowledge (Policy, Claims, Billing, Underwriting, Actuarial, Regulatory data)
- Experience with Azure DevOps (AzDO) and GitHub-based CI/CD
- Practical usage of Copilot for data engineering and development productivity
- Exposure to Generative AI / GenAI use cases in data platforms and analytics