Description

• Design, develop, and implement data solutions to meet business requirements and data ingestion needs facilitating accurate and timely data availability for analysis and decision-making.
• Extract, load, and Transform (ELT/ETL) data from various sources, including on-premises and cloud-based systems, APIs, databases, and files.
• Write well-designed, efficient code that adheres to security standards.
• Monitor and troubleshoot data pipelines to identify and resolve issues promptly to minimize disruptions in data processing for on-premises and cloud environments.
• Implement data quality checks and validation processes to ensure accuracy and completeness of data. collaborate with the data team to resolve them.
• Writes complex queries and scripts to efficiently manipulate, transform, and process raw data.
• Creates and executes data validation processes to ensure the reliability and consistency of incoming data. Build processes to monitor data quality.
• Continuously optimize data pipelines for performance, scalability, and reliability.
• Create and maintain technical documentation. Contribute to building and maintaining data catalog and lineage.
• Design and develop CI/CD processes that ensure high availability and agility.
• Develop cloud data services provisioning automation with integrated capabilities of IAM, network, security policies as code, and observability.
• Build tools and services to support data discovery, lineage, resiliency, and privacy compliance across the data platform.
• Stay updated with the latest trends and best practices in data engineering, cloud computing, and Azure services to suggest innovative solutions to continually improve the organization's data intelligence capabilities.
• Monitors and reports on supplier performance, customer satisfaction, adherence to security requirements and market intelligence.
• May mentor less experienced staff
• Performs other duties and tasks.
• May need to work outside of normal work hours (i.e., evenings and weekends)
• Travel may be required to other MTA locations or external sites.

• Strong knowledge of Big Data architectures, large data warehouses, and Data Lake solutions.
• Experience designing and implementing Modern Data and Analytics solutions including Lakehouse architecture and medallion architecture.
• Proficient in cloud services including Azure Databricks. Azure Synapse Analytics, Azure Data Factory, Azure Data Lake Store, Microsoft Purview, and Power BI.
• Experience deploying and running cloud-based data solutions and infrastructure-as-code frameworks like Terraform.
• Proficient in cloud deployments (MS Azure preferred) in an agile SDLC environment, leveraging modern programming languages, and DevOps.
• Experience with major database platforms including Oracle, SQL Server as well as Cloud databases and NOSQL Databases.
• Strong understanding of data engineering concepts, ELT/ETL principles, and data modeling.
• Experience with data integration techniques for both structured and unstructured data.
• Solid programming skills in languages such as Python, Pyspark, and SQL.
• Experience with Airflow.
• Experience in DevOps, Git Repos, and CI/CD pipelines for code deployment.
• Experience deploying and administering cloud-based data solutions using infrastructure-as-code and infra-automation tools like Terraform, Ansible, etc.
• Experience with Microsoft Purview is a plus.
• Strong knowledge of Microsoft Azure Cloud. AWS and GCP are desirable.
• Functional knowledge of Microsoft Power BI
• Experience with Jira, Confluence.
• Demonstrated ability to work independently and navigate organizational ambiguity.
• Effective written and verbal communication skills
• Hands-on programming experience in a business setting.
• Proficiency in at least one software engineering methodology, including but not limited to Agile, Scrum, DevOps, Extreme Programming (XP), Kanban, Lean, and Rapid Application Development (RAD).
• Experience applying structured validation and testing methods, including but not limited to Unit Testing, Integration Testing, System Testing, Acceptance Testing, and Regression Testing.
• Demonstrated ability to work independently and navigate organizational ambiguity.
• Effective written and verbal communication skills

• Education: Bachelor’s Degree 
• Experience: At least 3 years of relevant experience. An equivalent combination of education and experience may be considered instead of a degree. 

• Microsoft Certified: Azure Enterprise Data Engineer Associate 
• Microsoft Certified: Azure Data Fundamentals 

• Data Structures and Algorithms (Thorough Knowledge/Fully Proficient) 
• Cloud Computing (Thorough Knowledge/Fully Proficient) 

• Active Listening, Attention to Detail, Customer Service, 
• Prioritization, Problem-Solving, Effective Verbal and Written Communication