What are the responsibilities and job description for the IT Specialist (AI) position at National Institute of Standards and Technology (NIST)?
Summary
NIST works with industry and science to advance innovation and improve quality of life. We're looking for an IT Specialist (AI) to join our team! CAISI seeks a candidate to develop scientific methods and tools to rigorously evaluate AI systems in real-world settings. This notice is issued under direct-hire authority to recruit new talent to occupations for which NIST has a severe shortage of candidates.
Duties
In this position, you will: Evaluate and assess AI models in furtherance of the AI Action Plan. Design methods for scientifically sound evaluations of AI systems. Evaluate AI systems in application, post-deployment, and in real world settings. Develop methods, tooling, and guidelines for rigorous evaluation of end-to-end AI systems in use (e.g., in the workplace and/or in sector-specific tasks) to identify failure modes and areas of unreliability. Develop analyses of evaluations and AI measurement instruments themselves, for example, evaluation and improvement of red teaming, benchmarks, and automated scoring techniques. Adapt and improve evaluation techniques, and develop new techniques where needed. Collaborate with colleagues and across teams to develop innovative solutions to AI evaluation problems. Communicate complex AI concepts to both technical and non-technical audiences. Inform the development of technical guidelines and best practices for AI evaluation.
Qualifications
All applicants must submit a resume NOT TO EXCEED two pages on standard-size paper. If the resume exceeds two pages, only the first two pages will be reviewed for qualifications or assessment determinations. Basic Requirements: Experience must be IT related; the experience may be demonstrated by paid or unpaid experience and/or completion of specific, intensive training (for example, IT certification), as appropriate. For all positions individuals must have IT-related experience demonstrating each of the four competencies listed below. The employing agency is responsible for identifying the specific level of proficiency required for each competency at each grade level based on the requirements of the position being filled. Attention to Detail - Is thorough when performing work and conscientious about attending to detail. Customer Service - Works with clients and customers (that is, any individuals who use or receive the services or products that your work unit produces, including the general public, individuals who work in the agency, other agencies, or organizations outside the Government) to assess their needs, provide information or assistance, resolve their problems, or satisfy their expectations; knows about available products and services; is committed to providing quality products and services. Oral Communication - Expresses information (for example, ideas or facts) to individuals or groups effectively, taking into account the audience and nature of the information (for example, technical, sensitive, controversial); makes clear and convincing oral presentations; listens to others, attends to nonverbal cues, and responds appropriately. Problem Solving - Identifies problems; determines accuracy and relevance of information; uses sound judgment to generate and evaluate alternatives, and to make recommendations. ZP-2210-IV: In addition to the basic requirements, applicants must have one year (52 weeks) of specialized experience equivalent to at least the GS-12 level (ZP-III at NIST). Specialized experience is defined as experience demonstrating technical understanding of Artificial Intelligence (AI) generative models and Machine Learning (ML) algorithms; designing scientifically sound technical evaluation methods in real settings; carrying out rigorous technical evaluations to identify functionality and reliability issues; and collaborating and coordinating on AI-related projects. ZP-2210-V: In addition to the above requirements, applicants must have one year (52 weeks) of specialized experience equivalent to at least the GS-14 level (ZP-IV at NIST). Specialized experience is defined as experience demonstrating advanced technical understanding of Artificial Intelligence (AI) generative models and Machine Learning (ML) algorithms; designing scientifically sound technical evaluation methods in real settings; carrying out rigorous technical evaluations to identify functionality and reliability issues; and leading cross-functional AI-related teams and projects. Establishes best practices with specific technical standards and workflows. Experience refers to paid and unpaid experience, including volunteer work done through National Service programs (e.g., Peace Corps, AmeriCorps) and other organizations (e.g., professional; philanthropic; religious; spiritual; community, student, social). Volunteer work helps build critical competencies, knowledge, and skills and can provide valuable training and experience that translates directly to paid employment. You will receive credit for all qualifying experience, including volunteer experience. The qualification requirements in this vacancy announcement are based on the U.S. Office of Personnel Management (OPM) Qualification Standards Handbook. If requesting reconsideration of your qualification determination, please refer to the following site: Applicant Reconsideration
NIST works with industry and science to advance innovation and improve quality of life. We're looking for an IT Specialist (AI) to join our team! CAISI seeks a candidate to develop scientific methods and tools to rigorously evaluate AI systems in real-world settings. This notice is issued under direct-hire authority to recruit new talent to occupations for which NIST has a severe shortage of candidates.
Duties
In this position, you will: Evaluate and assess AI models in furtherance of the AI Action Plan. Design methods for scientifically sound evaluations of AI systems. Evaluate AI systems in application, post-deployment, and in real world settings. Develop methods, tooling, and guidelines for rigorous evaluation of end-to-end AI systems in use (e.g., in the workplace and/or in sector-specific tasks) to identify failure modes and areas of unreliability. Develop analyses of evaluations and AI measurement instruments themselves, for example, evaluation and improvement of red teaming, benchmarks, and automated scoring techniques. Adapt and improve evaluation techniques, and develop new techniques where needed. Collaborate with colleagues and across teams to develop innovative solutions to AI evaluation problems. Communicate complex AI concepts to both technical and non-technical audiences. Inform the development of technical guidelines and best practices for AI evaluation.
Qualifications
All applicants must submit a resume NOT TO EXCEED two pages on standard-size paper. If the resume exceeds two pages, only the first two pages will be reviewed for qualifications or assessment determinations. Basic Requirements: Experience must be IT related; the experience may be demonstrated by paid or unpaid experience and/or completion of specific, intensive training (for example, IT certification), as appropriate. For all positions individuals must have IT-related experience demonstrating each of the four competencies listed below. The employing agency is responsible for identifying the specific level of proficiency required for each competency at each grade level based on the requirements of the position being filled. Attention to Detail - Is thorough when performing work and conscientious about attending to detail. Customer Service - Works with clients and customers (that is, any individuals who use or receive the services or products that your work unit produces, including the general public, individuals who work in the agency, other agencies, or organizations outside the Government) to assess their needs, provide information or assistance, resolve their problems, or satisfy their expectations; knows about available products and services; is committed to providing quality products and services. Oral Communication - Expresses information (for example, ideas or facts) to individuals or groups effectively, taking into account the audience and nature of the information (for example, technical, sensitive, controversial); makes clear and convincing oral presentations; listens to others, attends to nonverbal cues, and responds appropriately. Problem Solving - Identifies problems; determines accuracy and relevance of information; uses sound judgment to generate and evaluate alternatives, and to make recommendations. ZP-2210-IV: In addition to the basic requirements, applicants must have one year (52 weeks) of specialized experience equivalent to at least the GS-12 level (ZP-III at NIST). Specialized experience is defined as experience demonstrating technical understanding of Artificial Intelligence (AI) generative models and Machine Learning (ML) algorithms; designing scientifically sound technical evaluation methods in real settings; carrying out rigorous technical evaluations to identify functionality and reliability issues; and collaborating and coordinating on AI-related projects. ZP-2210-V: In addition to the above requirements, applicants must have one year (52 weeks) of specialized experience equivalent to at least the GS-14 level (ZP-IV at NIST). Specialized experience is defined as experience demonstrating advanced technical understanding of Artificial Intelligence (AI) generative models and Machine Learning (ML) algorithms; designing scientifically sound technical evaluation methods in real settings; carrying out rigorous technical evaluations to identify functionality and reliability issues; and leading cross-functional AI-related teams and projects. Establishes best practices with specific technical standards and workflows. Experience refers to paid and unpaid experience, including volunteer work done through National Service programs (e.g., Peace Corps, AmeriCorps) and other organizations (e.g., professional; philanthropic; religious; spiritual; community, student, social). Volunteer work helps build critical competencies, knowledge, and skills and can provide valuable training and experience that translates directly to paid employment. You will receive credit for all qualifying experience, including volunteer experience. The qualification requirements in this vacancy announcement are based on the U.S. Office of Personnel Management (OPM) Qualification Standards Handbook. If requesting reconsideration of your qualification determination, please refer to the following site: Applicant Reconsideration