Senior Data Engineer

Regular Full-TimeFull Time

Stamford, CT, US

Salary: $185-190k

Position Overview:

The Principal Data Engineer holds the primary responsibility for skillfully designing, developing, implementing, and supporting the companies enterprise Microsoft Azure Data Warehouse. This role assumes accountability for ensuring the dependable, efficient, and secure operation and advancement of our On-Premise and Cloud Data Warehouses to effectively meet crucial business needs. As a lead figure in the organization, the candidate will manage intricate projects and offer technical leadership and guidance in all data-related matters for the Company.

Commitment to Internal Control:

The Senior Data Engineer is required to possess a comprehensive understanding of and adhere to the system of internal controls associated with the fundamental duties and responsibilities of the role. This includes compliance with SOX and all other pertinent regulatory and compliance policies and requirements.

Essential Duties and Responsibilities:

The responsibility of Principal Data Engineer encompasses the entire lifecycle of the Data Warehouse environments that underpin our vital business requirements. This includes the design, development, implementation, operation, and ongoing support of these critical systems.

The individual in this position is tasked with developing a flexible, enterprise-level environment that integrates multiple warehouses to guarantee precise, comprehensive, consistent, and timely data. Their primary goal is to create a cohesive system that fulfills these demands while catering to diverse business requirements.

The role necessitates the capacity to explore and grasp emerging technologies while collaborating closely with peer teams to establish strategic roadmaps and priorities. The ability to swiftly acquire and proficiently apply hands-on administration skills is essential. As a Subject Matter Expert in technical requirements, this position will play a crucial role in supporting and implementing data projects, as well as engaging effectively with users and other IT staff.

The Principal Data Engineer responsibilities include:

• Provide support in designing and overseeing enterprise-grade data pipelines and data stores, essential for the development of advanced analytics programs, machine learning models, and statistical methods.
• Implement automation and streamline processes to optimize the entire data and analytics platform, ensuring efficient throughput and high-performance outcomes.
• Recognize, devise, and execute internal process enhancements, including automation of manual tasks, optimizing data delivery, and redesigning architecture or infrastructure to enhance scalability.
• Collate large, intricate datasets that align with functional and non-functional business demands.
• Develop processes that facilitate data transformation, manage data structures, metadata, dependencies, and workload management.
• Collaborate with business users to understand functional and data requirements, contributing to the enhancement of data models and pipelines.
• Apply expert-level analytical and problem-solving skills to diagnose and resolve intricate technical issues.
• Create, maintain, and continuously enhance scalable data pipelines, while also developing new data source integrations to accommodate the growing volume and complexity of data.
• Designing, implementing, and managing data extraction, transformation, and loading (ETL) processes.
• Creating comprehensive technical specification documents and application interface designs.
• Creating data processing and integration solutions for both batch and real-time scenarios, proficiently handling structured and unstructured data.
• Leading and participating in design discussions, code reviews, and project-related team meetings.
• Providing mentoring and guidance to junior team members, promoting a culture of knowledge sharing and collaborative problem-solving.
• Ensuring data security and compliance with relevant regulations and best practices in all data operations.
• Provide assistance in troubleshooting and resolving data and system issues, stepping in when necessary to address outages and challenges.
• Other duties and responsibilities as assigned or needed.

KPI’s (Key Performance Indicators):

• Deliver Business Intelligence solutions that are 95% defect-free providing that adequate written business requirements, development time, and business test review were afforded during the project. This standard does not apply to legacy remediation efforts or ready to serve emergency production response activities.
• Effectively utilize consulting resources on all significant projects (over 40 hours) to allow for development power of scale. Consultants should do lower value work that is considered heavy lift, freeing up programmer analysts to spend more time in analysis and design while maintaining tight control over quality, code, and company intellectual property.
• These are overarching KPI metrics that are applicable to all goals that are defined over the course of the business year.

Responsibility and Decision-Making Authority:

• Exercise independent judgment and decision-making while adhering to Company Policy.

Management/Supervisory Responsibilities:

• Offers technical leadership to fellow IT staff members and actively engages with less experienced team members, providing valuable mentoring and guidance.
• Demonstrates capability in assuming a leadership role for managing projects.

Qualifications/Competencies:

Key Technical Knowledge, Skills, and Abilities:

• Exhibit a thorough understanding of Data Lake architectures, including raw, enriched, and curated layer concepts, and ETL/ELT operations.
• Exhibit a solid understanding of database design, data warehousing concepts, big data platforms, and ETL operations.
• Experience working with data integration techniques & self-service data preparation.
• Experience in requirements analysis, design, and prototyping.
• Experience deploying modern data solutions leveraging components like Azure functions, Azure Data Factory, Data Flows, Azure Data Lake, Azure SQL, Azure Synapse, Streaming Analytics or equivalent on another cloud provider such as AWS or Google Cloud.
• Experience with DevOps tools like Azure DevOps, Jenkins, Maven etc.
• Experience in building/operating/maintaining fault tolerant and scalable data processing integrations.
• Demonstrated experience of turning business use cases and requirements into technical solutions.
• Strong level of understanding on Azure Data Factory, SQL/Synapse, ADLS, and Azure DevOps.
• Ability to conduct data profiling, cataloging, and mappings for technical design and construction of data flows.
• Strong collaboration and experience working with remote teams.
• Strong problem-solving skills with emphasis on optimization data pipelines.
• Showcase excellent communication and presentation skills for effective collaboration with technical and non-technical stakeholders.
• Strong analytical skills and a drive to learn and master new technologies and techniques.
• Experienced using Azure Synapse Analytics. - MUST Have
• Experience working with third party providers and vendors for critical support requirements.

Competencies:

• Workload organization and prioritization
• Customer service - attitude and responsiveness
• Professional communications
• Strong project management skills as both a project leader and participant
• Good judgement, analytical capabilities and decision making.
• Able to work independently and as part of a small, tightly integrated team.
• Good documentation skills and attention to detail

Education and Experience:

• Bachelor’s degree with a minimum of 10 years of related experience.
• 10 years of hands-on Data Warehouse architecture and development experience
• Demonstrated expertise in Microsoft SQL Server and/or Azure development.

Working Hours / Travel Requirements:

• Hours are 5 days a week onsite and may vary and will require periodic overtime, including occasional evening and weekends, depending on business needs.
• On call 24x7 for emergency support
• Occasional travel for business meetings, seminars or training may be required.

Physical Demands:

• Digital dexterity and hand/eye coordination in operation of office equipment.
• Light lifting and carrying of supplies, files, etc.
• Ability to speak to and hear customers and/or other employees via phone, person or virtually.
• Body motor skills sufficient to enable incumbent to move from one office location to another.

Salary: $185-190k per year, plus a discretionary Bonus.

The salary range is determined and based on internal equity, market data/ranges, applicant's skills, prior relevant experience, and education.

Additional Benefits:

• Medical, Dental and Vision Plans

• 401(k) and matching

• Generous Paid Time Off

• Company paid Life Insurance

• Employee assistance program

• Training and Development Opportunities

• Employee discounts