What are the responsibilities and job description for the Azure Databricks Engineer - Azure AI Foundry position at Acunor Infotech?
Azure Databricks Engineer - Azure AI Foundry
Acunor is seeking an Azure Databricks Engineer to build scalable Azure data platforms and integrate enterprise data with Azure AI Foundry for analytics, machine learning, and Generative AI applications.
Responsibilities- Design, develop, and optimize Azure Databricks data pipelines and modern lakehouse architectures.
- Build ingestion frameworks for structured and unstructured sources; develop ETL/ELT using PySpark, Spark SQL, and Delta Lake.
- Integrate enterprise data platforms and knowledge repositories with Azure AI Foundry to support AI and GenAI use cases.
- Prepare, transform, and manage training datasets in collaboration with data scientists and AI engineers.
- Develop and manage Azure Data Factory workflows.
- Implement data quality, validation, monitoring, governance, security, and privacy controls.
- Apply CI/CD and DevOps practices to data engineering delivery; collaborate with architects and business stakeholders.
- 8-10 years of relevant experience.
- Hands-on Azure Databricks, PySpark, Spark SQL, Delta Lake, ADLS Gen2, Azure Data Factory, Azure Synapse Analytics, Azure SQL Database, and data modeling.
- Strong Python and SQL programming skills.
- Experience with Azure Storage Services, Key Vault, Monitor, Functions, and Logic Apps.
- Azure DevOps, Git, CI/CD pipelines, and Infrastructure as Code experience.
- Experience working with Azure AI Foundry and Azure AI Services, including data pipelines that feed AI and GenAI applications.
- Understanding of LLMs, RAG, vector databases, embeddings, and prompt engineering; knowledge of Azure OpenAI integration patterns.
- Exposure to AI model lifecycle management and monitoring; experience integrating enterprise knowledge repositories with AI applications.
- Understanding of AI governance, security, and responsible AI principles.
- Bachelor's or Master's degree in Computer Science, Engineering, Data Science, Information Technology, or a related field.
- Scala and Terraform.
- Microsoft Fabric; Azure AI Search, Pinecone, or Chroma.
- MLflow and MLOps frameworks; Kubernetes and containerization.
- Spark Structured Streaming or Kafka; conversational AI or GenAI use cases.
- Relevant Microsoft Azure or Databricks certifications are nice to have.