Backend Software Engineer (Platform)

Backend Software Engineer (Platform)

W2 Contract

Pay Rate: $55 - $65 per hour

Location: Cupertino, CA - Hybrid Role

Job Summary:

We are seeking strong Software Engineers to support onboarding and metadata ingestion into our Data Registry, a central data catalog platform. In this role, you will work on the systems and connectors that enable organizations across our organization to register datasets into the Data Registry through scalable, reliable ingestion pipelines. This position is ideal for an engineer who is comfortable working across data ingestion, connector development, operational troubleshooting, and platform support in a distributed data ecosystem.

Duties and Responsibilities:

• Build, enhance, and support metadata ingestion pipelines for onboarding datasets into a large Data Registry
• Develop and maintain connector-based ingestion workflows using open source DataHub connector patterns
• Troubleshoot failed ingestions, analyze logs, identify root causes, and restore pipeline health
• Work with internal customers to support both pull-based and push-based onboarding models
• Modify existing connectors and build custom Python and Scala-based ingestion components when required
• Support operational monitoring and ongoing reliability of ingestion services
• Investigate connectivity, deployment, and interoperability issues across distributed environments
• Collaborate with partner teams to onboard new data sources and improve onboarding efficiency
• Contribute to evolving ingestion architecture, including batch and selected real-time ingestion patterns
• Support platform transitions and deployment improvements across the onboarding ecosystem

Requirements and Qualifications:

• 5 years of software engineering experience, or 3 years with a master's degree
• Strong programming experience in Python and Scala
• Experience designing, building, or supporting ETL, ingestion, or metadata pipeline workflows
• Hands-on experience debugging production issues in distributed systems
• Strong understanding of logging, monitoring, and operational troubleshooting
• Experience working with data systems, connectors, or pipeline orchestration in cloud or hybrid environments
• Ability to write clean, maintainable, well-tested code and operate effectively in a fast-paced engineering environment
• Strong communication and collaboration skills

Preferred Qualifications:

• Experience with open source DataHub, especially connectors or metadata ingestion frameworks
• Working knowledge of Scala
• Experience with batch and streaming data technologies such as Spark, Kafka, or Flink
• Familiarity with monitoring and observability platforms such as Datadog
• Understanding of network connectivity concepts, including ingress, egress, and cross-system integration
• Experience building custom connectors or integration components for niche source systems
• Familiarity with deployment and onboarding patterns in modern data platforms
• Exposure to AWS services and data governance tooling, such as Lake Formation
• Computer Science degree or equivalent, strong foundational background in software engineering

Desired Skills and Experience

Backend Engineering, Platform Engineering, Data Ingestion Pipelines, Metadata Management, ETL Development, Data Pipeline Architecture, Data Registry Systems, Data Catalog Platforms, Python, Scala, Connector Development, DataHub, Distributed Systems, Pipeline Troubleshooting, Log Analysis, Root Cause Analysis, Monitoring & Observability, Datadog, Cloud & Hybrid Environments, Batch Processing, Real-Time Streaming, Apache Spark, Apache Kafka, Apache Flink, Data Integration, API & Connector Frameworks, System Interoperability, Deployment & DevOps Support, Network Connectivity (Ingress/Egress), AWS, Lake Formation, Data Governance, Pipeline Orchestration, Software Development Best Practices, Code Quality & Testing, Cross-functional Collaboration

Bayside Solutions, Inc. is not able to sponsor any candidates at this time. Additionally, candidates for this position must qualify as a W2 candidate.

Bayside Solutions, Inc. may collect your personal information during the position application process. Please reference Bayside Solutions, Inc.'s CCPA Privacy Policy at www.baysidesolutions.com.