Responsibilities

• Lead the technical operations including the architect, design, expansion, monitoring, support, and maintenance for Scientific Computing’s computational and data science ecosystem consistent with best practices. Key components include a 50,000 core and 30 petabyte usable high-performance computing cluster, clinical data warehouse and software development environment.
• Lead the troubleshooting, isolation and resolution of all technical issues
• Lead the design, development, implementation and management of all system administration tasks, including hardware and software configuration, configuration management, system monitoring (including the development and maintenance of regression tests), usage reporting, system performance (file systems, scheduler, interconnect, high availability, etc.), security, networking and metrics, etc.
• Ensures that the design and operation of the HPC ecosystem is productive for research.
• Collaborates effectively with research and hospital system IT, compliance, HIPAA, security and other departments to ensure compliance with all regulations and Sinai policies.
• Partners with other peers regionally, nationally and internationally to discover, propose and deploy a world-class research infrastructure for Mount Sinai.
• Prepares and manages budgets for hardware, software and maintenance. Participates in chargeback/fee recovery analysis and provides suggestions to make operations sustainable.
• Lead the integration of HPC resources with laboratory equipment such as genomic sequencers, etc.
• Researches, deploys and optimizes resource management and scheduling software and policies and actively monitoring.
• Designs, tunes, manages and upgrades parallel file systems, storage and data-oriented resources.
• Researches, deploys and manages security infrastructure, including development of policies and procedures.
• Lead and assist the team to resolve user support requests from researchers.
• Assists in developing and writing system design for research proposals.
• Lead the development of a framework for effective system documentation.
• Works effectively and productively with other team members within the group and across Mount Sinai.
• Provide after-hours support in case of a critical system issue.

Qualifications

• Bachelor’s degree in computer science, engineering or another scientific field. Master's or PhD preferred.
• 8 years of progressive HPC system administration and operations (preferably in a Redhat/CentOS Linux administration, Batch HPC cluster environment)
• Must be an expert troubleshooter; Must be a team player and customer focused
• Strong experience with configuration management systems such as xCAT, Puppet and/or Ansible
• Strong experience with networking and security
• Strong experience with Infiniband and Gigabit Ethernet
• Experience with LSF and GPFS Spectrum Scale parallel file systems and storage
• Experience with providing technical operations leadership
• Ability to manage a variety of disparate tasks and priorities independently and troubleshoot complex technology problems.
• Attention to detail; time and project management skills.
• Excellent communication skills, analytical ability, strong judgment and management skills, and the ability to work effectively as a liaison between both research and technology teams.
• Strong written, oral, and interpersonal communication skills
• Script and programming experience

Preferred Experience

• Experience with archival storage and tape libraries (TSM) is highly preferred.
• Experience with databases and web services is highly preferred.
• Compliance, HIPAA, GDPR, FISMA
• Experience with managing web access to HPC resources (such as Open OnDemand)
• Experience in a research environment is highly preferred.
• Experience with financial budgets and providing cost benefit analysis is preferred.
• Cloud Technology

Job Type: Full-time

Benefits:

• 401(k)
• Dental insurance
• Health insurance

Ability to Commute:

• Andes, NY 13731 (Required)

Ability to Relocate:

• Andes, NY 13731: Relocate before starting work (Required)

Work Location: In person