Senior Principal Platform Engineer (K8s)

Clarity Innovations is a trusted national security partner, dedicated to safeguarding our nation’s interests and delivering innovative solutions that empower the Intelligence Community (IC) and Department of Defense (DoD) to transform data into actionable intelligence, ensuring mission success in an evolving world.

Our mission-first software and data engineering platform modernizes data operations, utilizing advanced workflows, CI/CD, and secure DevSecOps practices. We focus on challenges in Information Warfare, Cyber Operations, Operational Security, and Data Structuring, enabling end-to-end solutions that drive operational impact.

We are committed to delivering cutting-edge tools and capabilities that address the most complex national security challenges, empowering our partners to stay ahead of emerging threats and ensuring the success of their critical missions. At Clarity, we are people-focused and set on being a destination employer for top talent, offering an environment where innovation thrives, careers grow, and individuals are valued. Join us as we continue to lead innovation and tackle the most pressing challenges in national security.

Position Summary

The position is part of a team of software and platform engineers building a unified, multi-tenant control plane driven by Kubernetes, Crossplane, and Cluster API (CAPI), that abstracts infrastructure differences across AWS, Azure, on-premises, and air-gapped environments.
As a Senior Principal Platform Engineer (Services), you will bridge the gap between development needs and production deployment environments. Instead of just managing vendor charts, you will package, configure, and extend foundational services running on Kubernetes. Utilizing Go and Python, you will automate delivery workflows and serve as the technical authority on Kubernetes networking, identity, secrets, policy, observability, and secure supply chain requirements.
The ideal candidate is a highly independent engineer capable of driving complex initiatives from inception to completion with minimal supervision. You bring strong software engineering practices to automate, validate, and scale repeatable deployments, combined with a passion for modernizing delivery systems, strengthening secure supply chains, and leveraging AI to accelerate high-quality engineering work.

Key Responsibilities 

Kubernetes Services Packaging & Integration

• Platform Service Packaging & Integration: Package, configure, and seamlessly integrate foundational Kubernetes services, such as Keycloak, Vault, Istio, Kyverno, OPA, Grafana, Alloy, Mimir, and Loki, ensuring clean interoperability across identity, secrets management, service mesh, policy enforcement, and observability systems.
• Packaging Standards & Interfaces: Define reusable packaging patterns, schemas, validation rules, and automation that frontend and other teams can build on when enabling users to create bundles and packages.
• OCI Image & Bundle Management:  Build, validate, and maintain OCI images, deployment bundles, Helm charts, and Kubernetes manifests with a focus on reproducibility, versioning, and reliable delivery across deployment environments, including air-gapped, disconnected, on-premise systems.
• Software-Driven Automation: Use Go, Python, and CI/CD automation to validate service behavior, reduce manual deployment steps, and improve the reliability of service delivery.

Secure Supply Chain & Engineering Excellence

• Secure Software Supply Chain: Establish and improve secure supply chain practices for third-party application packaging, including artifact provenance, vulnerability management, dependency tracking, signing, and validation.
• CI/CD Automation: Build and maintain GitLab CI pipelines that automate packaging, testing, scanning, validation, promotion, and publishing of service artifacts.
• Deployment Reliability: Define validation, health check, rollback, and upgrade patterns that make service deployments repeatable and reliable across cloud, on-premises, and air-gapped environments.
• Maintain Architecture Records: Contribute to and execute Architecture Decision Records (ADRs) regarding service packaging standards, deployment patterns, security controls, and platform integration decisions.
• Cross-Team Technical Leadership: Partner with and mentor engineers across frontend, backend, control plane, and infrastructure teams to help them adopt service packaging patterns, understand Kubernetes deployment constraints, and operate within a software-defined, control-plane-first approach to Kubernetes cluster management.

Qualifications 

• Advanced Kubernetes Engineering: 5 years of hands-on experience deploying, scaling, and troubleshooting production services, with deep expertise in control loops, reconciliation behavior, and extension patterns (CRDs, Operators, webhooks, admission policies) that goes beyond standard managed services (EKS/AKS).
• Service Packaging & Lifecycle Management: Deep experience packaging, configuring, upgrading, and maintaining Kubernetes-based services using OCI images, Helm charts, Kubernetes manifests, and deployment bundles, with attention to versioning, compatibility, rollback, and long-term maintainability.
• Infrastructure Automation & CI/CD: Proficiency with GitLab CI, GitOps workflows, FluxCD, and declarative configuration tools such as Helm and Kustomize to automate service packaging, validation, promotion, and deployment.
• Linux, Containers & Networking: Strong understanding of Linux, container runtimes, OCI images, DNS, TLS, ingress, service mesh concepts, network policy, and Kubernetes networking needed to troubleshoot services across cloud, on-premises, and air-gapped environments.
• Secure Supply Chain Practices: Experience with vulnerability scanning, dependency management, SBOMs, artifact provenance, image signing, and secure promotion workflows for third-party software and containerized services.
• Platform Services Experience: Hands-on experience with three or more categories of foundational platform services, such as identity and access management, secrets management, observability, service mesh, policy enforcement, or admission control. Relevant technologies may include Keycloak, Vault, Grafana, Alloy, Mimir, Loki, Istio, Kyverno, OPA, or similar systems.
• Software Engineering Fundamentals: Strong professional software engineering experience, including Go and/or Python, automated testing, code review, debugging, and maintainable automation for platform delivery workflows.
• Systems Thinker: Ability to view services, infrastructure, Kubernetes primitives, deployment workflows, and team boundaries as parts of one platform system rather than isolated components.
• Defensive Engineering Mindset: An engineering approach that assumes distributed systems and network partitions will fail asynchronously. You naturally design highly available, self-healing control planes with robust fallback mechanisms.
• Complex Systems Debugger: An investigative mindset capable of debugging complex Kubernetes and distributed service failures where resources, controllers, network dependencies, certificates, secrets, and startup order interact in subtle ways.
• Extreme Ownership: A proven track record of acting as a technical steward for a service, platform capability, or subsystem, where you care as much about operational metrics, edge-case failures, maintainability, and technical debt as you do about delivering new functionality.

Preferred Qualifications 

• Air-Gapped or Regulated Environments: Experience delivering software into disconnected, restricted, regulated, or high-security environments.
• Production Operations: Hands-on experience supporting production services, troubleshooting incidents, and improving reliability based on operational feedback.
• Telemetry-Driven Debugging: Advanced capability using metrics, logs, traces, dashboards, alerts, and Kubernetes events to diagnose service and platform issues.
• Service Lifecycle Ownership: Experience managing long-term service lifecycle concerns, including version upgrades, compatibility testing, deprecation planning, and repeatable release processes.
• AI-Accelerated Engineering: Demonstrated ability to use AI tools to accelerate engineering work, investigate unfamiliar systems, generate or improve tests and documentation, and increase delivery speed while maintaining strong technical judgment.

Salary Range: $120,000 - $345,000