What are the responsibilities and job description for the Data Engineer Python, React, Databricks & Azure position at SANS?
Job Title Data Engineer Python, React, Databricks & Azure Location Hybrid (New York, NY / Edison, NJ) Duration 12 Months
Job Overview: We are seeking an experienced Data Engineer with strong hands-on expertise in React JS, Core Python, Azure, Databricks, and Delta Lake to design, develop, and support scalable data engineering solutions. The ideal candidate will build Python-based ETL pipelines, develop data workflows for GenAI applications, implement data quality frameworks, and collaborate with cross-functional teams to deliver reliable data platforms.
Key Responsibilities
- Design, develop, and maintain Python-based ETL/ELT pipelines for enterprise data ingestion, transformation, validation, and storage.
- Develop cloud-native data solutions using Azure Function Apps, Azure Blob Storage, Azure Cosmos DB, Azure Databricks, and Delta Lake.
- Build and optimize Databricks notebooks, workflows, Delta Tables, and data transformation pipelines using Python and PySpark.
- Develop reusable Core Python modules, libraries, utilities, and automation frameworks for data engineering processes.
- Build data workflows supporting GenAI portals and applications, ensuring reliable ingestion, processing, storage, and availability of data.
- Develop and maintain data pipelines integrating structured and semi-structured data from multiple sources.
- Implement automated data validation, reconciliation, completeness, and quality checks to identify missing, duplicate, or incomplete records.
- Optimize Databricks and Python workloads for performance, scalability, reliability, and cost efficiency.
- Work with React to support data-driven portal components, dashboards, data visualization, and integration with backend APIs.
- Develop and integrate REST APIs and backend services to expose processed data to React-based applications.
- Implement monitoring, logging, error handling, alerting, and recovery mechanisms across data pipelines.
- Provide production support, including incident triage, root cause analysis, troubleshooting, problem management, and SLA adherence.
- Collaborate with business stakeholders, vendors, developers, QA teams, and application teams to understand requirements and deliver enhancements.
- Participate in code reviews, technical design discussions, testing, deployment, and release activities.
- Maintain technical documentation for data pipelines, architecture, data flows, operational procedures, and support processes.
Required Skills & Qualifications
- Strong hands-on experience with Core Python and Python-based data engineering.
- Experience with Azure cloud services, particularly:
- Strong experience with Databricks, PySpark, Delta Lake, and Delta Tables.
- Experience designing and maintaining ETL/ELT pipelines and data workflows.
- Hands-on experience implementing data quality and validation frameworks.
- Experience with React.js and integrating React applications with backend APIs/data services.
- Strong understanding of REST APIs, JSON, API integration, and microservices-based architectures.
- Experience with Git-based source control and CI/CD processes.
- Strong SQL skills and experience working with relational and NoSQL data sources.
- Experience with application monitoring, production support, incident management, and troubleshooting.
- Strong communication and collaboration skills with technical and non-technical stakeholders.
Preferred Skills
- Experience supporting GenAI/AI platforms or data portals.
- Knowledge of Azure DevOps and CI/CD pipelines.
- Experience with Azure API Management.
- Knowledge of data lakehouse architecture and Medallion Architecture.
- Experience with Agile/Scrum development methodologies.
- Familiarity with observability, logging, and cloud monitoring tools.
Thanks and regards,
Amit Lakhotia