Manager, Psychometric Operations

The American Board of Internal Medicine (ABIM) is currently seeking a Manager, Psychometric Operations to join our team. In this role, the Manager, Psychometric Operations, leads psychometric operations and methodologies supporting the development, scoring, and reporting of ABIM examinations. As a core member of the Psychometrics leadership team, this role contributes to department-level strategic decision-making, including staffing, resource planning, and budget considerations. The position oversees and improves psychometric processes related to item analysis, calibration and equating, scoring, and results reporting. The role works closely with physicians and other stakeholders in activities such as item writing, standard setting, and score reporting. This position directly supervises multiple doctoral-level professional staff and represents the department and organization in interactions with external partners and vendors.

Reporting to the Director, Psychometric Operations has the following primary responsibilities:

• Develop and lead assessment scoring, evaluation, and results reporting initiatives that support the development, delivery, and continuous improvement of current and future ABIM assessments.
• Oversee operational aspects of item analysis, calibration, equating, and examination scoring processes. Evaluate examination processing workflows and implement improvements to enhance efficiency, accuracy, and quality control.
• Lead psychometric analyses for at least one assessment, including attending item review meetings with content experts, conducting item analyses, performing scoring and equating, and facilitating standard-setting meetings and analyses.
• Supervise and provide technical leadership to psychometric staff across all functions, including exam scoring and score reporting, scoring system design and maintenance, and participation in organization-wide projects.
• Prepare and present examination results and related research findings to physician and non-physician audiences, and respond to inquiries related to examination scoring and manage communications with the public regarding examination processing and assessment data.
• Plan, manage, and oversee projects proactively by anticipating workload, staffing, timelines, and resource needs, while ensuring effective and responsible use of organizational fiscal and human resources.
• Collaborate with Assessment & Research management staff to lead examination creation, scoring, and score reporting initiatives to ensure accuracy, timeliness, and adherence to established standards.
• Provide strategic direction and oversight to the Director, Psychometric Operations, and the Vice President of Psychometrics on operational assessment programs.

We are seeking a highly skilled and experienced professional with a doctoral degree and at least 5 years of relevant work experience in professional or educational testing or operational assessment, including supervisory experience. The ideal candidate will have a strong background in managing assessment projects, leading teams, and applying best practices in testing, with a solid foundation in psychometric theory and practice, including item response theory, classical test theory, equating, scaling, standard setting, and generalizability theory; a strong understanding of high-stakes assessment policies and operations; and experience in research design and statistical analysis. They will be proficient in Microsoft Office Suite (Word, PowerPoint, and Excel) and statistical software (SAS and/or R). Comfort with PC/Windows-based systems and basic word processing is also required.

The successful candidate will demonstrate excellent organizational, planning, and problem-solving skills, with the ability to manage multiple projects, meet deadlines, and adapt to shifting priorities. Strong written and verbal communication skills are essential, as well as the ability to build professional relationships with external customers, ABIM staff, and Board members. They should have experience managing projects end-to-end, with strategic thinking and sound decision-making based on analysis and facts. The candidate must quickly grasp ABIM policies and communicate them effectively. A commitment to confidentiality and integrity, along with flexibility and a willingness to learn new skills, is essential. Periodic U.S. travel for vendor meetings and conferences is required. This is an exciting opportunity for a detail-oriented, results-driven professional to contribute to high-quality testing and assessment initiatives in a fast-paced environment.

This role is based in our offices at 510 Walnut Street, Philadelphia, PA, with the ability to work remotely three days a week. Our technology and facility teams provide and support a seamless hybrid work environment for all employees.

***

At the American Board of Internal Medicine (ABIM), our team of unique and talented professionals, and the inspiring work they do, are essential to meeting the needs and expectations of ABIM’s diverse community of physicians.

ABIM is like no other workplace, and we are proud of it. We are committed to recognizing the importance of our people by investing in their lives through ongoing learning opportunities and exceptional total compensation & benefit offerings.

We look forward to learning more about your interest in joining our team. EOE