What are the responsibilities and job description for the Sr Data Engineer + PySpark, Python position at Virtusa?
We are seeking a highly experienced and motivated Senior Data Engineer to join our team. The ideal candidate will have extensive experience in designing, building, and optimizing highly scalable and robust ETL/ELT pipelines. This role will be critical in shaping our data architecture, implementing Lakehouse solutions, and working with cutting-edge technologies like LLMs and Vector Search.
Key Responsibilities
Key Responsibilities
- Design, develop, and maintain robust and scalable data pipelines using PySpark and Python.
- Implement and manage data solutions within the Databricks platform.
- Define and enforce data modeling standards, specifically utilizing the Medallion architecture (Bronze, Silver, Gold layers).
- Architect and implement Lakehouse capabilities, including AI/Machine Learning features and Vector Search for advanced data retrieval.
- Evaluate, integrate, and work with Large Language Model (LLM) frameworks.
- Collaborate with data scientists and business stakeholders to understand data requirements and translate them into technical solutions.
- Ensure data quality, reliability, and security throughout the data lifecycle.
- Mentor junior engineers and contribute to best practices in data engineering.