What are the responsibilities and job description for the Sr. Data Engineer position at SoTalent?
Sr. Data Engineer
π Location: Shelton, Connecticut, United States
π’ Industry: Restaurants
πΌ Work Setting: Hybrid
Are you passionate about designing modern data platforms, building scalable data pipelines, and enabling enterprise-wide analytics through cloud-based technologies?
We are seeking an experienced Senior Data Engineer to lead the design, development, and optimization of data integration and analytics solutions that power critical business operations and decision-making.
In this role, you will collaborate with product owners, business stakeholders, and engineering teams to deliver reliable, scalable, and high-performing data pipelines. You will play a key role in supporting enterprise data initiatives, implementing data governance standards, and leveraging modern cloud data platforms to create trusted, analytics-ready data assets.
Key Responsibilities
Data Pipeline Development
- Design, develop, and maintain scalable, high-performance data pipelines and integrations.
- Build reliable data solutions that support both operational system integrations and enterprise analytics platforms.
- Ensure data pipelines are resilient, flexible, and aligned with enterprise architecture standards.
System Integration & Project Delivery
- Partner with cross-functional teams to deliver enterprise integration initiatives.
- Collaborate with stakeholders to gather requirements, define technical solutions, and estimate development efforts.
- Ensure successful delivery of large-scale data and integration projects.
Automation & DevOps
- Develop automated testing frameworks and deployment processes for data pipelines and integrations.
- Support CI/CD practices to improve deployment reliability and development efficiency.
- Contribute to continuous improvement of engineering standards and automation capabilities.
Data Quality & Governance
- Ensure high levels of data quality, consistency, and accuracy across enterprise data platforms.
- Implement data governance, master data management, lineage tracking, and metadata management best practices.
- Conduct data analysis and troubleshooting to resolve data issues and support business needs.
Operational Support
- Provide advanced technical support for critical data and integration solutions.
- Assist operational teams with issue resolution and root cause analysis.
- Monitor data platform performance and proactively address potential challenges.
Documentation & Knowledge Sharing
- Create and maintain technical documentation, including data flows, lineage diagrams, architecture diagrams, and operational procedures.
- Participate in code reviews and provide mentorship to junior engineers.
- Promote engineering best practices and knowledge sharing across teams.
Required Qualifications
- Bachelor's degree in Computer Science, Information Systems, Engineering, or a related field, or equivalent professional experience.
- 5β8 years of experience designing and building enterprise data pipelines and system integrations.
- Minimum 3 years of experience working within cloud-based data environments.
- Strong experience delivering scalable, production-ready data engineering solutions.
- Excellent communication and stakeholder management skills.
Technical Skills
Cloud Data Platforms
- Hands-on experience building modern data solutions using cloud-based data platforms such as Databricks, Snowflake, or equivalent technologies.
- Experience implementing lakehouse architectures and modern analytical data ecosystems.
Data Engineering
- Expertise in ETL/ELT development, data ingestion, transformation, and orchestration.
- Experience developing robust batch and streaming data pipelines.
- Knowledge of modern workflow orchestration and pipeline automation tools.
Data Architecture & Modeling
- Experience designing layered data architectures for analytics-ready data.
- Strong understanding of dimensional modeling, data vault methodologies, and analytical schema design.
- Ability to create scalable models that support reporting, analytics, and AI initiatives.
Programming & Querying
- Strong proficiency in SQL for complex data analysis and transformation.
- Experience with Python and/or PySpark for data engineering and automation development.
- Ability to optimize data processing and improve solution performance.
Performance Optimization
- Experience tuning large-scale data workloads and optimizing processing efficiency.
- Knowledge of partitioning strategies, query optimization, clustering techniques, and resource management.
Governance & Security
- Familiarity with enterprise data governance frameworks, access controls, data lineage, and security best practices.
- Experience implementing governed data assets and maintaining data compliance standards.
DevOps & CI/CD
- Experience using version control systems and modern deployment frameworks.
- Knowledge of automated testing, release management, and infrastructure-as-code principles.
Cloud Ecosystem
- Experience working with cloud services supporting data engineering workloads across AWS, Azure, GCP, or similar environments.
Analytics & AI Readiness
- Understanding of data platforms that support advanced analytics, machine learning, and generative AI use cases.
- Experience preparing trusted datasets for analytical and AI-driven solutions.
Preferred Competencies
- Strong analytical and problem-solving skills.
- Ability to work effectively in cross-functional and agile environments.
- Experience mentoring junior engineers and conducting code reviews.
- Strong attention to detail and commitment to data quality.
- Ability to manage multiple priorities in a fast-paced environment.
- Excellent verbal and written communication skills.
What Success Looks Like
- Delivering scalable and reliable enterprise data pipelines.
- Ensuring high-quality, trusted data across analytical platforms.
- Supporting critical business initiatives through modern data engineering solutions.
- Driving adoption of data governance and engineering best practices.
- Enabling advanced analytics, reporting, and AI capabilities through well-designed data architectures.