What are the responsibilities and job description for the Azure Data Lead (Azure, ETL, ADF & Databricks) position at Lorven Technologies, Inc.?
Hi ,
Our client is looking for an Data Lead (Azure, ETL, ADF & Databricks) for a project and below is the detailed requirement.
Job Title: Data Lead (Azure, ETL, ADF & Databricks)
Location: Dallas, TX
Job Description:
- Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field with 13 years of overall IT experience.
- 7–10 years of experience in Data Engineering and ETL Development.
- 5 years of hands-on experience with Azure Data Services.
- Design and support Azure-based data solutions leveraging: Azure Data Factory (ADF), Azure Databricks, Azure Data Lake Storage, Azure Synapse Analytics, Azure SQL Database, Azure Key Vault, Azure Functions
- Strong experience developing enterprise ETL/ELT solutions.
- Design, develop, and maintain scalable ETL/ELT pipelines using Azure Data Factory (ADF).
- Build and optimize data ingestion frameworks for structured, semi-structured, and unstructured data.
- Develop reusable data transformation frameworks and data quality solutions.
- Implement efficient batch and near-real-time data processing pipelines.
- Develop data transformation and processing workflows using Azure Databricks.
- Build scalable solutions utilizing PySpark, Spark SQL, and Delta Lake.
- Optimize Spark jobs for performance, reliability, and cost efficiency.
- Implement data cleansing, enrichment, aggregation, and validation processes.
- Data Warehousing & Analytics: Design and maintain dimensional data models and enterprise data warehouses.
- DevOps & Automation: Implement CI/CD pipelines using Azure DevOps. Data Governance & Security.