Sr. Data Engineer

We are seeking an experienced Sr. Engineer, Data – Privacy to design, develop, and optimize scalable data architectures and enterprise-grade data pipelines across cloud, on-premises, and hybrid environments. The ideal candidate will possess deep expertise in Azure Data Services, Databricks, Snowflake, Python, and SQL, along with hands-on experience in implementing privacy-focused data engineering solutions and leveraging AI-powered development tools.

This role will partner closely with data engineers, architects, and business stakeholders to deliver high-performing, secure, and compliant data platforms that enable data-driven decision-making at scale.

Key Responsibilities

• Design and develop scalable data pipelines using Azure Data Factory (ADF) for integrating data from multiple sources.
• Build and optimize data ingestion, transformation, and processing frameworks using Snowflake, including:
• Stored Procedures
• Streams
• Tasks
• Snowpipe
• Dynamic Tables
• Iceberg Tables
• Storage Integrations and Views
• Design and manage enterprise data solutions using Azure Databricks, Apache Spark, and Unity Catalog.
• Develop and optimize solutions on Azure Data Lake Storage (ADLS) for secure and high-performance data storage.
• Create efficient and scalable SQL queries, stored procedures, and data models.
• Monitor, troubleshoot, and optimize data pipelines for performance, reliability, and cost efficiency.
• Implement data quality frameworks, validation mechanisms, and monitoring processes.
• Ensure data governance and compliance with privacy and regulatory requirements.
• Collaborate with cross-functional teams to deliver scalable and innovative data solutions.
• Mentor team members and contribute to engineering best practices and technical excellence.

AI & Automation Responsibilities

• Utilize AI productivity tools such as Claude, Cursor, or similar IDEs for:
• Code generation
• SQL development
• Debugging
• Documentation
• Code reviews
• Testing
• Design and implement AI agents for privacy data engineering solutions.
• Apply Foundation Models, Prompt Engineering, and Retrieval-Augmented Generation (RAG) techniques for data engineering use cases.
• Implement observability, audit logging, and human-in-the-loop controls for AI-assisted workflows.

Required Qualifications

• Bachelor's degree in Computer Science, Computer Engineering, or a related field.
• 5–7 years of hands-on experience in Data Engineering and ETL development.
• Strong experience with:
• Azure Data Factory (ADF)
• Azure Databricks
• Azure Data Lake Storage (ADLS)
• Snowflake
• Oracle
• Python
• SQL
• Experience building and migrating enterprise data solutions on Microsoft Azure.
• Strong expertise in SQL, NoSQL, and relational database design.
• Experience with Azure Flink, dbt, and open-source data frameworks.
• Hands-on experience with CI/CD implementation using Azure DevOps or GitLab.
• Working knowledge of Kafka and stream processing technologies.
• Excellent analytical, problem-solving, communication, and stakeholder management skills.
• Ability to manage multiple priorities in a fast-paced environment.

Preferred Qualifications

• Experience with Foundation Models, Prompt Engineering, RAG, and AI Agent Development.
• Microsoft Azure certifications such as:
• DP-203: Azure Data Engineer Associate
• Snowflake Certification
• Databricks Certification
• Experience with Power BI reporting and visualization.

ABOUT BRICKRED SYSTEMS

BrickRed Systems is a global leader in next-generation technology consulting and workforce solutions, specializing in delivering high-quality talent across digital,engineering, marketing, analytics, finance, operations, and business transformation domain With a strong emphasis on innovation, scalability, and client success, BrickRed Systems helps organizations solve complex business challenges by providing skilled professionals across strategy, technology, creative, and operational functions. BrickRed fosters a culture of continuous learning, collaboration, and excellence, enabling professionals to contribute to high-impact global initiatives while advancing their careers.