What are the responsibilities and job description for the Azure Data Engineer with AI Fabric position at Wall Street Consulting Services LLC?
Job Title: Azure Data Engineer with AI Fabric
Location: Warren, NJ (Onsite)
Domain: Insurance
Experience: 10
- Microsoft Fabric & lakehouse engineering Hands-on production experience with Microsoft Fabric One Lake, Lakehouse, Data Factory pipelines, Spark notebooks, and Direct Lake mode for Power BI. Must have built and operated something real on Fabric Fluent in PySpark and Delta Lake: MERGE, schema evolution, time travel, OPTIMIZE, partitioning. Has built incremental ingestion from ADLS Gen2 or S3 into curated Delta tables. Strong SQL (window functions, CTEs, query optimization) and a working knowledge of Parquet internals, partitioning, predicate pushdown, and compaction.
- Event-driven ingestion on Azure Has built production ingestion using Azure Event Hubs
- Python engineering & Azure platform fundamentals Modular Python code with type hints, unit tests, and packaging. Git-based workflow with pull requests and CI. Working knowledge of ADLS Gen2, Azure Active Directory, Key Vault, and Managed Identity permissions and the engineer need to handle them confidently from day one.
- Source-to-target data modeling with a canonical layer Has built or contributed to a canonical model EDP, data vault, or dimensional that decouples source systems from downstream consumers. Understands medallion architecture (Bronze / Silver / Gold) and can explain why each layer exists and what belongs in it.
- Production LLM & embedding experience using an LLM API (Azure OpenAI strongly preferred for our stack) with structured output via JSON mode or function calling, paired with an embedding model for semantic search or matching.