What are the responsibilities and job description for the Cribl Engineer Expert position at DAn Solutions?
REQUIRES AN EXISTING/ACTIVE TS/SCI WITH CI POLYGRAPH - NO REMOTE WORK, MUST WORK ON SITE
Job Description:
We are seeking a highly experienced Cribl Engineer to serve as the principal technical authority for observability pipelines built on Cribl Stream and Cribl Edge. This role is designed for a senior technologist with deep expertise in log/telemetry routing, largescale data engineering, and enterprise-grade observability architectures.
You will shape pipeline strategy, design complex routing and transformation logic, drive platform reliability, mentor senior engineers, and serve as the top technical escalation point for Cribl-related challenges.
What You'll Do
- Lead architecture and design for Cribl Stream/Edge across multiple enclaves and data domains.
- Build high throughput pipelines (multiTB/day) with advanced routing, filtering, enrichment, and replay workflows.
- Optimize system performance, worker topology, CPU/memory distribution, queues, and transport mechanisms.
- Engineer secure data flows with masking, tokenization, RBAC, PKI/TLS, and other governance controls.
- Integrate pipelines with SIEM/analytics ecosystems (Splunk, Elastic, SaaS telemetry platforms, cloud services).
- Develop HA/DR patterns, reliability frameworks, fleet health metrics, and failure mode response processes.
- Maintain reusable Cribl packs, shared patterns, runbooks, and operational standards.
- Serve as the senior escalation point for Cribl issues; interface with vendor engineering as required.
- Mentor engineers, conduct design reviews, drive engineering excellence, and enforce architectural standards.
- Support cross functional teams (security, cloud, analytics, infrastructure) on logging and telemetry strategy.
Required Qualifications
- 10 years of experience in logging, observability, or SIEM engineering.
- 5 years architecting enterprise scale log/telemetry pipelines.
- 3 years hands‑on with Cribl Stream and Cribl Edge in production environments.
- Demonstrated success operating and scaling pipelines at 5–10 TB/day.
- Expert-level experience with Splunk forwarding/ingestion, source type management, and indexing practices.
- Strong Linux fundamentals; scripting expertise (Python/Bash); Git; automation (Ansible/Terraform).
- Strong understanding of transport protocols (HTTP, TCP, TLS/MTLS), Kafka, S3/object storage.
- Experience designing secure data flows, including encryption, RBAC, secrets management, and compliance controls.
- Demonstrated ability to mentor senior engineers and lead technical decision making.
- Certified Cribl Certified Engineer (CCOE) or equivalent Cribl product expertise.
- Must possess the following DoD 8570.01-M certifications or be willing to obtain within 30 days of hire:
- Information Assurance Technician (IAT) Level II certification (currently Security CE, CCNA-Security, GSEC, SSCP, CySA , GICSP, or CND).
- IAT Level III certification requirements (currently CASP CE, CCNP Security, CISA, CISSP (or Associate), GCED, or GCIH).
- Cyber Security Service Provider (CSSP) - Infrastructure Support (IS) certification requirements (currently CEH, CySA , GICSP, SSCP, CHFI, CFR, Cloud , or CND).
- Must possess a TS/SCI with Polygraph
Preferred Qualifications
- Expertise creating and maintaining Cribl Packs and reusable pipelines.
- Experience with cloud telemetry (AWS, Azure, hybrid) and cross‑domain data movement patterns.
- Familiarity with NIST / CIS control frameworks and secure engineering practices.
- Experience building observability frameworks for large distributed systems.
- Vendor engagement experience (Cribl PS, product teams, troubleshooting escalations).