Senior Data Engineer

Position Title: Data Engineer

Work Location: Coopersburg, PA (Hybrid)

Type of Employment: Full time

Role Overview:

We are seeking a highly skilled Data Engineer to develop and manage high volume and complex data pipelines using AWS cloud services and multiple ERP platforms like SAP, Oracle applications, Salesforce etc.

In this role, you will contribute to the overall data architecture, migration strategy, governance model, and implementation best practices for a scalable and secure solutions. You will collaborate with business and technical teams to design solutions that enable advanced analytics, self-service reporting, and enterprise data adoption.

Key Result Areas and Activities:

1. Architect & Design

• Define and lead the design of enterprise-scale data platforms, including ingestion pipelines, transformation frameworks, and data models for structured and unstructured data.
• Drive modernization initiatives by migrating legacy systems to cloud-native architectures (AWS).
• Architect scalable, secure, and high-performance solutions leveraging distributed systems, data lakes, and data warehouses.
• Establish standards for data modeling, metadata management, and semantic layers to enable advanced analytics and reporting.

2. Development Responsibilities

I: Software Asset Maintenance & Upgrades

• Upgrade and maintain third-party components such as ingress controllers, service meshes, monitoring agents, infrastructure libraries, and cloud-native tools.
• Apply the latest versions and security patches to ensure compliance, stability, and performance.

II: Infrastructure as Code (IaC) Enhancements

• Update and enhance IaC scripts to support version upgrades across development, QA, and production environments.
• Validate changes through sandbox testing before deployment to production.

III: Compatibility & Dependency Management

• Ensure upgraded components remain compatible with dependent services and applications.
• Identify and mitigate potential breaking changes or dependency conflicts.

IV: Application Code Adjustments

• Implement necessary code changes in supported languages (e.g., Python) to accommodate new versions or configuration requirements.
• Address minor and moderate changes required for compatibility with upgraded components.
• Update existing unit tests in response to the application code changes.

V: Security & Compliance

• Apply immediate fixes for vulnerabilities.
• Maintain adherence to organizational security and governance guidelines.

VI: Testing & Validation

• Create and execute test strategies for validating the upgrades.
• Execute existing unit tests and manual test cases post-upgrade.
• Conduct functional testing of impacted applications to ensure end-to-end stability.
• Validate application behavior after code changes and infrastructure updates.

VII: Reporting & Governance

• Provide weekly status reports detailing software versions, security posture, upgrade activities, and testing outcomes.
• Participate in regular reviews and acceptance processes.

3. Provide Advisory and Consulting Services

• Act as a trusted advisor to business and technology leaders on data strategy, governance, and architecture.

4. Team Collaboration and Leadership

• Mentor and guide engineering teams on best practices for data engineering, ETL design, and performance optimization.

Must-Have Skill Set:

• Expertise in designing and implementing large-scale data architectures (data lakes, warehouses, streaming platforms).
• Deep understanding of cloud data services (AWS) and migration strategies.
• Strong proficiency in ETL/ELT pipelines and framework development using Python.
• Skilled in building and optimizing data pipelines for batch and real-time processing.
• Strong knowledge of data governance, security (row-level, column-level), and compliance frameworks.
• Advanced SQL and experience with multiple relational and NoSQL databases (Must have - Redshift, RDS and Athena).
• Experience with data visualization platforms like Denodo.
• Experience with building scheduled jobs (data load, cache and individual query) in Denodo.
• Ability to interpret complex stored procedures and optimize queries for performance.
• Familiarity with Agile methodologies and DevOps practices for data engineering.
• Exceptional communication skills for engaging executives and non-technical stakeholders.
• Knowledge of containerization (Docker, Kubernetes) and orchestration for data workloads.
• Experience with BI tools integration with enterprise data platforms.

Good-to-Have Skill Set:

• Certifications in cloud platforms (AWS) and data engineering.
• Experience with advanced analytics and machine learning pipelines.
• Prior consulting experience or leading large-scale data transformation programs.
• Knowledge of data extraction from SAP OData Services.

Qualification:

• Bachelor’s degree in computer science, engineering, or related field (master's degree is a plus).
• Demonstrated continued learning through one or more technical certifications or related methods.
• At least 10 years of relevant experience; two years may be substituted for a master’s degree.