Senior Systems Engineer #339

The Senate Sergeant at Arms is seeking a Senior Systems Engineer. The complete vacancy announcement and application can be found on the United States Senate Career Page at https://sen.gov/1VNZQ. This vacancy announcement closes at 7pm EST on the closing date. Late applications will not be accepted.

Qualifications:

Other Duties/ Responsibilities

• Participates in on-site and on-call rotation.

Required Work Experience

• Five to eight years of relevant work experience (GED), three to six years (Associates), one to three years (Bachelor’s), one to two years (Master’s).

Required Special Skills/ Knowledge

• As part of our hiring process, we may conduct a skills assessment to better understand an applicant’s proficiency in key areas relevant to the role. 
• Manages current storage capacity and ensures storage is available to support all computing environments; plans for future storage growth; monitors and confirms that system files and application products are successfully backed up; performs necessary installations, patches, and upgrades.
• Works escalated issues from lower-tier team members. Mentors/relays information on solutions to train others.
• Analyzes new hardware and software releases for their impact on the Senate environment; tests new releases on Senate supported platforms; develops product configuration standards for use in the Senate environment.
• Creates, develops, modifies, instructs and provides technical assistance on business continuity plans and processes.

Desired Qualifications 

• Extensive experience serving in a Splunk administration or engineering role withina complex enterprise environment.
• Demonstrated experience building, implementing, and administering distributed Splunk environments on Red Hat Enterprise Linux (RHEL) operating systems, including versions 9, and 10.
• Hands-on experience supporting clustered Splunk architectures, including heavy forwarders, indexers, and search head clusters, with responsibilities spanning security, performance optimization, and operational support.
• Proven ability to edit, maintain, and optimize Splunk configuration files and applications in production environments.
• Demonstrated experience onboarding and normalizing data into Splunk using forwarders, deployment servers, scripted inputs, TCP/UDP inputs, and modular inputs from diverse data sources.
• Experience developing and integrating application interfaces using REST APIs.
• Proficiency in scripting and automation using technologies such as Ansible,Python, and JavaScript, including the use of automation frameworks to manage infrastructure and deployments.
• Demonstrated experience operating across multiple technology stacks, applying sound engineering principles beyond a single platform, and integrating Splunk with complementary enterprise systems.
• Experience evaluating, implementing, or supporting emerging technologies in support of operational, security, or data analytics initiatives.
• Proven ability to identify opportunities for technical innovation and translate them into practical, scalable solutions aligned with organizational objectives.

Working Conditions

• The U.S. Senate network cannot be taken offline for maintenance during the workday, or while the Senate is in session. As such, maintenance windows may only occur at night, on weekends, and occasionally on holidays. Employees who perform systems upgrades, maintenance, wiring, backups, support our alternate data centers will have schedules that include working nights, weekends, and holidays.
• This position directly supports essential services of the U.S. Senate. As such, this position requires the employee be available and prepared to work during a lapse, in inclement weather, on holidays, weekends, and during late nights to ensure essential services to the Senate continue without interruption. In the context of government furloughs, this position is considered excepted.
• Since this position requires onsite presence in support of the U.S. Senate when in session, this position requires the employee to be available and prepared to work during a lapse, in inclement weather, on holidays, and during late night, overnight, and weekend sessions. In the context of government furloughs, this position is considered excepted. 

Security Clearance

• This position requires that the applicant obtain and maintain a Secret U.S. Government security clearance.
• Applicants must be U.S. citizens in order for the SAA to submit your application for a security clearance.

Responsibilities:

Contributes to the section's work effort in responding to incidents from other sections, including the help desk. Supports tasks which may include helping to develop standard operating procedures, assignments, and coordination of work efforts to address issues. Assists users with computer needs such as installing and configuring software/hardware on their PC; troubleshoots problems with user systems. Provisions new accounts for users. Supports the section's work effort to develop processes and procedures to streamline the resolution of issues. Provides technical support for Senate information technology or telecommunications services, systems, hardware, software, and storage solutions to the users; supports Senate offices, contract support vendors, and other technical staff in troubleshooting and managing the various Senate environments. Maintains and monitors LAN/WAN and SAN operations and solves LAN/WAN and SAN applications problems; installs integrates, customizes, enhances, troubleshoots, and tests LAN and SAN hardware and software. Assists in installing, upgrading, customizing, tuning, and maintaining databases for optimal performance and availability; assists in developing, testing, and implementing database store procedures, functions, triggers, views, and packages. Conducts acceptance and integration testing; assists vendor systems implementation by troubleshooting related system issues affecting Senate users. Analyzes, designs, develops, implements, and maintains applications to meet the needs of Senate users; assists with product evaluations and implementation. Attends internal training provided by subject matter experts to understand operational tasks, documents support needs and share the information within the assigned section. Other duties as assigned.

Salary : $102,745