Psychometrician

Job Summary:

The Psychometrician is responsible for designing, developing, analyzing, and scoring The American Board of Pediatrics (ABP) assessment instruments that evaluate the competencies of pediatricians. The role applies psychological principles and statistical methods to ensure the reliability, validity, and fairness of ABP assessments. Core activities include collaborating with ABP staff and subject matter experts regarding exam development, scoring, and reporting; meeting facilitating; and conducting research to provide actionable insights for improving certification assessments and processes.

Essential Duties and Responsibilities:

• Provides consultation regarding the design, development, and use of tests and tools in exam design processes that assess the competencies of pediatricians.
• Conduct studies to identify the knowledge and skills required for pediatricians and use study results to develop exam specifications.
• Establishes passing standards using appropriate psychometric methods to ensure valid score inferences and/or classification decisions.
• Performs a wide variety of psychometric analyses, including analyzing test items to ensure they are functioning as intended and free from bias, and analyzing test forms to ensure consistency and fairness.
• Ensures score comparability across different test forms and over time, allowing for fair assessment of all test-takers.
• Develops and implements scoring processes and algorithms.
• Designs and generates score reports that provide meaningful feedback to test-takers and other stakeholders.
• Conducts psychometric research to improve assessment instruments and practices and to support evidence-based decision-making.
• Summarizes and conveys complex technical information (e.g., procedural methods, analysis results) in an accurate and understandable manner, either verbally or in writing.
• Works collaboratively with other team and staff members and provides education as needed to ensure the validity and fairness of certification processes, tools, and resulting decisions.
• Manages large datasets/databases and ensures data integrity.

Other Functions:

• Contributes expertise to professional organizations, participates in conferences, mentors emerging professionals, and supports initiatives that advance the field of psychometrics.
• Supports ad-hoc assessment department or ABP activities (eg, facilitating test development meetings, serving on special internal committees).
• Performs other duties as assigned.

Supervisory Responsibilities:

No formal supervisory responsibilities are required, but the psychometrician may be expected to provide mentorship to the psychometric graduate student as needed.

Required Qualifications:

• PhD in educational measurement and statistics, educational psychology, or a closely related field.
• Graduate-level coursework in measurement theory, classical test theory, item response theory, test equating, and quantitative and qualitative research methods.
• Proficiency using SAS, R, and Winsteps for data manipulation and analysis tasks.
• Two (2) years of experience conducting item-level and exam-level analyses using both Classical Test Theory (CTT) and Item Response Theory (IRT) models/methods.
• Two (2) years of experience creating psychometric documents and reports (e.g., statistical summary reports, reports summarizing procedural methods and results, policy and procedure manuals, professional manuscripts, and presentations).
• Two (2) years of experience conducting psychometric research, including study design, data collection, data analysis, and dissemination of results.
• One (1) year of experience planning and facilitating exam-related meetings or workshops (e.g., practice analysis, standard setting, item development).
• Skill in conveying complex, technical, or quantitative information to non-technical audiences.
• Strong communication skills, both written and verbal.
• Time management skills, including the ability to set priorities and a strong commitment to meeting deadlines.
• High attention to detail.
• Ability to collaborate effectively as the specific project/task demands.
• Ability to perform essential duties and responsibilities (listed above) with minimal supervision.

Preferred Qualifications:

• Two (2) years of psychometric experience in a professional licensure or certification setting.
• Coursework and one year of experience in advanced statistical modeling (eg, multivariate modeling, factor analysis, multi-level modeling, structural equation modeling).
• Basic proficiency in database management, including one year of experience working with relational tables and developing protocols for ensuring data integrity.
• One (1) year of experience creating graphics, charts, and other data visualizations.
• One (1) year of project management experience.

Travel Required

Travel is expected to be between 5 - 10% of the time.  

Work Environment:

The work environment is an office setting. The noise level in the work environment is usually low to moderate. The environment is also temperature-controlled. This may vary if working from home.

Physical Demands:

While performing the duties of this job, the employee is regularly required to communicate with others. The employee is frequently required to stand, walk, sit, demonstrate hand and finger dexterity in the course of their work, handle or feel, and reach with hands and arms. The employee is occasionally required to climb, balance, stoop, kneel, crouch or crawl.

Reasonable accommodations may be made to enable individuals with disabilities to perform the essential duties and responsibilities.

The expected physical workload is expected to be sedentary work - exerting up to 10 pounds of force occasionally and/or a negligible amount of force frequently or constantly to lift, carry, push, pull, or otherwise move objects, including the human body. Sedentary work involves sitting most of the time. Jobs are sedentary if walking and standing are required only occasionally and all other sedentary criteria are met.

This role is expected to use the following machines, tools, and equipment:

·        Computer and associated equipment (eg, external monitors, mouse, printer, keyboard)

·        Virtual meeting tools and software

·        Statistical analysis software

·        Meeting room AV systems