Observability Engineer – DataDog / AppDynamics

Job Title: Observability Engineer – DataDog / AppDynamics SME

Location: Scottsdale, AZ

Work Model: Hybrid – 2 days onsite per week (mandatory)

Interview Process: Onsite interview required

Job Summary

We are seeking an experienced Observability Engineer / Monitoring SME with strong hands-on expertise in DataDog, AppDynamics, and Quantum Metrics. The ideal candidate will have a solid background in full-stack observability, application performance monitoring (APM), monitoring automation, and Site Reliability Engineering (SRE).

This role is primarily hands-on, with approximately 70% focused on observability platform administration, configuration, troubleshooting, and automation, and 30% dedicated to SRE-related initiatives involving Prometheus, OpenTelemetry (OTel), Grafana, and related technologies.

The successful candidate will collaborate with application, infrastructure, and engineering teams to design and implement effective monitoring solutions, improve alerting strategies, and reduce operational toil through automation.

Key Responsibilities

• Administer, configure, customize, deploy, and troubleshoot DataDog and AppDynamics agents across application and infrastructure environments.
• Manage and optimize Quantum Metrics monitoring capabilities.
• Implement and maintain native mobile monitoring, browser real-user monitoring (RUM), and synthetic monitoring solutions.
• Analyze application performance, metrics, logs, distributed traces, and service dependencies to identify and resolve performance issues.
• Design, develop, and implement monitoring strategies, dashboards, alerts, and observability solutions aligned with business and technical requirements.
• Collaborate with application, cloud, infrastructure, and SRE teams to gather monitoring requirements and implement effective solutions.
• Develop and maintain automation for routine operational tasks using Python and GitOps practices.
• Support and improve observability capabilities in Google Cloud Platform (GCP) and Kubernetes environments.
• Contribute to SRE initiatives involving Prometheus, OpenTelemetry (OTel), Grafana, and related monitoring technologies.
• Identify opportunities to leverage AI and automation to reduce alert noise, manual intervention, and operational toil.
• Troubleshoot monitoring gaps, agent deployment issues, instrumentation challenges, and alerting inefficiencies.
• Establish best practices for monitoring configuration, platform administration, automation, and observability standards.

Required Qualifications

• 5 years of experience in full-stack observability, including APM tools, metrics collection, log analysis, and distributed tracing.
• 3 years of strong experience working with observability platforms, including DataDog, AppDynamics, and Quantum Metrics.
• Hands-on expertise in DataDog and AppDynamics agent deployment, configuration, customization, maintenance, and troubleshooting.
• Practical experience implementing native mobile monitoring, browser real-user monitoring (RUM), and synthetic monitoring.
• Strong experience designing and developing automation for repetitive operational tasks using Python and GitOps.
• 3 years of experience with Google Cloud Platform (GCP).
• Solid understanding of Kubernetes and cloud-native application environments.
• Strong knowledge of application performance monitoring, infrastructure monitoring, metrics, logs, traces, dashboards, and alerting strategies.
• Ability to work with cross-functional teams to gather requirements, identify monitoring gaps, and implement scalable observability solutions.
• Strong analytical and troubleshooting skills, with a focus on automation and continuous improvement.
• Must be able to work onsite in Scottsdale, AZ, 2 days per week.
• Must be available for an onsite interview.

Preferred Qualifications

• Experience leveraging AI to improve automation, reduce alert noise, and minimize operational toil.
• Experience with Dynatrace, Prometheus, OpenTelemetry (OTel), and Grafana.
• Familiarity with SRE principles, reliability engineering, incident response, and service performance optimization.
• Experience implementing observability best practices in cloud-native and distributed application architectures.

Role Breakdown

• 70% – Observability Platform Administration and Automation: DataDog, AppDynamics, Quantum Metrics, agent deployment, monitoring configuration, troubleshooting, and Python/GitOps automation.
• 30% – Site Reliability Engineering: Prometheus, OpenTelemetry, Grafana, and related reliability and monitoring initiatives.

Work Location and Interview Requirements

This is a hybrid position based in Scottsdale, Arizona, requiring onsite presence 2 days per week. The onsite arrangement is mandatory, and candidates must be willing to attend an in-person interview.