What are the responsibilities and job description for the Data Engineer position at Techgene Solutions?
Job Description
Note: Looking for an experienced Data Engineer to build and maintain Azure-based data pipelines using Databricks, PySpark, and Python. The candidate should have strong Azure Data Factory experience, work with large datasets, and support ETL development and data validation.
7 years of experience required.
Top 3 Mandatory Skills and Experience • Agile development/5 years • Databricks, PySpark, Python/3-5 yrs • Azure cloud Data Factory/3 yr • Data Validations/3-5 yrs
How will the resource(s) fit into your team?: Azure, ETL-Databricks
- 4 years of proven working experience as a Data Engineer
- 3 years of experience with Python, pyspark
- 2 years of experience in Microsoft Azure Data Explorer, DataBricks, Data Lake and (PowerBI or Tableau)
- 3 years full-stack development experience, preferably in Azure
- 3 years of cloud development and data lake experience (prefer Microsoft Azure) including Azure EventHub, Azure Data Factory, Azure Functions, ADX, ASA, Azure Databricks, Azure DevOps, Azure Blob Storage, Azure Data Lake, Azure Power Apps and Power BI.
- Technical expertise regarding data models, database design development, data mining and segmentation techniques
- Knowledge of statistics and experience using statistical packages for analyzing datasets (Excel, SPSS, SAS, etc)
Must Have Skills
- Databricks
- PySpark
- Python
- Azure Data Factory
- Azure Data Lake
- Data Validation
- Agile Development