What are the responsibilities and job description for the Alert Management & Observability Standards Lead position at Jobs via Dice?
Dice is the leading career destination for tech experts at every stage of their careers. Our client, Avenue Code, LLC, is seeking the following. Apply via Dice today!
Alert Management & Observability Standards Lead
Fairfield CA
Job Description
Job Title: Alert Management & Observability Standards Lead Role Summary The Alert Management & Observability Standards Lead is responsible for rationalizing and governing all system alerts to ensure they align with department priorities, operational coverage models, and service reliability goals. This role defines alerting standards, reviews and approves alerts before they are routed to the 24x7 Eyes-on-Glass Operations team, and establishes a scalable approach to cataloging alert response instructions (runbooks/playbooks) so responders can take consistent, high-quality actions. This position operates at the intersection of the IT Operations Command Center (OCC), engineering/application teams, platform/monitoring tool owners, and service owners, ensuring alerts are actionable, prioritized, and paired with clear response guidance. Key Responsibilities 1) Alert Rationalization & Prioritization (Core) Establish and maintain a department-wide alert rationalization framework that evaluates alerts for: Business/service criticality and operational priority Actionability (clear operator action available) Signal-to-noise (duplicate/low-value alerts removed or suppressed) Ownership and escalation paths Perform regular alert reviews (new existing) to ensure alert quality, correct routing, and alignment with operational coverage. Lead continuous improvement efforts to reduce alert fatigue while preserving detection of true incidents and high-impact degradation. 2) Standards, Policies, and Guardrails Define and enforce alerting standards including: Severity definitions and thresholds Required metadata (service, CI, owner, runbook link, escalation) Naming conventions and tagging taxonomy Routing rules and “when to page vs. when to ticket” Create a standardized Alert Design Checklist and approval workflow (e.g., “Definition of Done” for alert onboarding). Partner with tool/platform owners to ensure standards are embedded in monitoring tooling (templates, required fields, automated validation). 3) Routing Decisions to 24x7 Eyes-on-Glass Act as gatekeeper (or lead the governance process) for determining which alerts should: Go to 24x7 Eyes-on-Glass for immediate triage Route to on-call engineering directly Create tickets for business-hours handling Be suppressed, aggregated, or converted to dashboards/health indicators Ensure routing aligns with: Operational responsibilities and skills of the Eyes-on-Glass team Department priorities (e.g., safety, reliability, customer impact) Service ownership and support models 4) Runbook / Response Instruction Cataloging (Knowledge System) Establish a consistent approach to cataloging response instructions for every actionable alert, including: “What does this alert mean?” (symptoms impact) “What to check first” (triage steps) “What actions to take” (standard remediation) “When to escalate and to whom” (clear escalation triggers) Links to dashboards, logs, SOPs, and known issues Own the runbook template and ensure runbooks are versioned, maintained, and reviewed on a defined cadence. Partner with service owners to ensure runbooks stay current as systems change. 5) Reporting & Operational Outcomes Define and publish KPIs that demonstrate alerting health and operational performance, such as: Alert volume trends by service and severity Percentage of alerts with runbooks and valid ownership Alert “actionability rate” and noise reduction Mean time to acknowledge / triage effectiveness (as applicable) Facilitate governance forums (weekly/monthly) with service owners and engineering leads to review alert quality and backlog. 6) Cross-Functional Enablement Coach service teams on best practices: SLIs/SLOs, alert thresholds, dependency monitoring, and incident correlation. Drive adoption of observability patterns (golden signals, health indicators, multi-signal alerting). Support major incident learning by feeding post-incident insights back into:
Note
Alert Management & Observability Standards Lead (req ending in 4328)
Alert Management & Observability Standards Lead
Fairfield CA
Job Description
Job Title: Alert Management & Observability Standards Lead Role Summary The Alert Management & Observability Standards Lead is responsible for rationalizing and governing all system alerts to ensure they align with department priorities, operational coverage models, and service reliability goals. This role defines alerting standards, reviews and approves alerts before they are routed to the 24x7 Eyes-on-Glass Operations team, and establishes a scalable approach to cataloging alert response instructions (runbooks/playbooks) so responders can take consistent, high-quality actions. This position operates at the intersection of the IT Operations Command Center (OCC), engineering/application teams, platform/monitoring tool owners, and service owners, ensuring alerts are actionable, prioritized, and paired with clear response guidance. Key Responsibilities 1) Alert Rationalization & Prioritization (Core) Establish and maintain a department-wide alert rationalization framework that evaluates alerts for: Business/service criticality and operational priority Actionability (clear operator action available) Signal-to-noise (duplicate/low-value alerts removed or suppressed) Ownership and escalation paths Perform regular alert reviews (new existing) to ensure alert quality, correct routing, and alignment with operational coverage. Lead continuous improvement efforts to reduce alert fatigue while preserving detection of true incidents and high-impact degradation. 2) Standards, Policies, and Guardrails Define and enforce alerting standards including: Severity definitions and thresholds Required metadata (service, CI, owner, runbook link, escalation) Naming conventions and tagging taxonomy Routing rules and “when to page vs. when to ticket” Create a standardized Alert Design Checklist and approval workflow (e.g., “Definition of Done” for alert onboarding). Partner with tool/platform owners to ensure standards are embedded in monitoring tooling (templates, required fields, automated validation). 3) Routing Decisions to 24x7 Eyes-on-Glass Act as gatekeeper (or lead the governance process) for determining which alerts should: Go to 24x7 Eyes-on-Glass for immediate triage Route to on-call engineering directly Create tickets for business-hours handling Be suppressed, aggregated, or converted to dashboards/health indicators Ensure routing aligns with: Operational responsibilities and skills of the Eyes-on-Glass team Department priorities (e.g., safety, reliability, customer impact) Service ownership and support models 4) Runbook / Response Instruction Cataloging (Knowledge System) Establish a consistent approach to cataloging response instructions for every actionable alert, including: “What does this alert mean?” (symptoms impact) “What to check first” (triage steps) “What actions to take” (standard remediation) “When to escalate and to whom” (clear escalation triggers) Links to dashboards, logs, SOPs, and known issues Own the runbook template and ensure runbooks are versioned, maintained, and reviewed on a defined cadence. Partner with service owners to ensure runbooks stay current as systems change. 5) Reporting & Operational Outcomes Define and publish KPIs that demonstrate alerting health and operational performance, such as: Alert volume trends by service and severity Percentage of alerts with runbooks and valid ownership Alert “actionability rate” and noise reduction Mean time to acknowledge / triage effectiveness (as applicable) Facilitate governance forums (weekly/monthly) with service owners and engineering leads to review alert quality and backlog. 6) Cross-Functional Enablement Coach service teams on best practices: SLIs/SLOs, alert thresholds, dependency monitoring, and incident correlation. Drive adoption of observability patterns (golden signals, health indicators, multi-signal alerting). Support major incident learning by feeding post-incident insights back into:
Note
Alert Management & Observability Standards Lead (req ending in 4328)
- Goal: Rationalize and reduce alert noise for the 24x7 NOC; establish monitoring standards and thresholds across compute, network, and application layers
- Key tools in use: Comarch OSS, Spectrum OI, NetBrain, NetMRI, Dynatrace, SCOM — Splunk was recently removed
- Work split: ~85-90% hands-on technical, 10-15% governance
- Ideal profile:
- Empathy toward 24x7 NOC and emergency response environments
- Ability to translate technical alert data into business impact language
- Comfortable working with pushback — Joe will provide executive backing
- Work arrangement: Hybrid — 1 to 2 days/week on-site; local candidates preferred near Fairfield, CA (Sacramento area also acceptable)
- Reporting: Direct report to Joe, cross-functional across all teams
- Schedule: No 24x7 shift requirements for this role
- Equipment: Supplier provides laptop; candidate logs in via VDI (Joe will try to request a PG&E laptop when possible)