Job Details

This ad is expired.

Psychometrician & Program Evaluator
Go to our website »

Position Number: 2563176
Location: Baltimore, MD
Position Type: Laboratory and Research

Psychometrician & Program Evaluator
We are seeking a Psychometrician & Program Evaluator to be a part of the Office of Assessment and Evaluation (OAE).

The mission of OAE is to enhance the educational programs in the Johns Hopkins University School of Medicine (JHUSOM) through evidence-based best practices in the assessment of learners and the evaluation of programs. OAE partners with JHUSOM faculty, staff, and learners to design, launch, and maintain assessment and evaluation initiatives in graduate biomedical education, postdoctoral education, undergraduate medical education, graduate medical education, and continuing medical education. OAE also supports educational activities initiated by the Simulation Center, the Institute for Excellence in Education, the Office of Medical Student Affairs, the Office of Faculty Development, and the Professional Development and Career Office.

As Part of it's Core Mission, OAE:

Supports the assessment of learners and the evaluation of programs through the selection, design, vetting, and implementation of assessment and evaluation tools (e.g., rubric design, multiple-choice question exam development, survey design, interview guide development)
Facilitates understanding of the assessment of learners and the evaluation of programs through the analysis, reporting, and use of related data (e.g., statistical analysis of quantitative data, thematic content analysis of qualitative data)
Conducts faculty development activities in areas related to learner assessment and program evaluation (e.g., validity, reliability, exam blueprints, standard setting, test and item analysis)

As a secondary aim, OAE also provides consultation in research design, statistical analysis and interpretation, and manuscript preparation intended for scholarly reports in medical and biomedical education, as projects relate to assessment and evaluation initiatives at JHUSOM.

This role reports to the Director of the OAE and works closely with the OAE team and other stakeholders, including faculty, staff, and learners.

The primary purpose of the role is to design and implement psychometric analysis for educational testing programs and support educational program evaluation. This requires competencies in psychometric methods, assessment design, qualitative and quantitative analysis.

Specific Duties & Responsibilities:

Duties and responsibilities include, but are not limited to:

General
- Align assessment and evaluation practices and activities with learner needs, program goals, accreditation standards, and evidence-based best practices.
- Collaborate with faculty members, program administrators, and other stakeholders to design, launch, and maintain assessment and evaluation activities.
- Oversee and implement assessment and evaluation plans.
- Plan, coordinate, and perform complex assessment and program evaluation assignments.
- Accurately analyze, interpret, and present data results to diverse audiences.
- Work with faculty on developing research studies and provide analysis and guidance.
- Maintain professional and effective working relationships with JHUSOM educational program stakeholders and with partnering units, including the JHUSOM Office of Information Technology, Office of Medical Student Affairs, Office of Medical Student Admissions, Office of Graduate Biomedical Education, Office of Graduate Medical Education, Office of Postdoctoral Affairs, and Office of the Registrar.
- Serve as an active member of various committees as needed.
- Coordinate with the Precision Education Data Governance Specialist on the collection, management, and access of educational data stored in a data lake environment.
- Other duties as assigned.
Psychometrics
- Collaborate with content area experts to continually inform and improve current assessments and / or develop new assessments.
- Conduct and lead item and test analysis, equating, standard setting, score reporting, analysis, reliability and validity studies and other psychometric activities.
- Provide psychometric services to support faculty and staff (e.g., educate and support faculty in conducting appropriate statistical / assessment analyses).
- Be accountable for basic and complex statistical design and analysis decisions such as variance estimations (weighting), sample design, statistical modeling, and psychometrics.
- Participate in the design, development, maintenance, and continuous improvement of assessments.
- Provide support for assessment management systems, institutional databases, and data analysis.
Evaluation
- With the Director of OAE, help develop plans to provide learning assessment, learner outcomes (both during and after their time at JHUSOM), program reviews, and teaching evaluation for JHUSOM medical and biomedical education and initiatives that support and continually advance educational excellence.
- Assist the Director of OAE with program evaluation improvement strategies to advance day-to-day operations.
- Use program evaluation models including Logic Models to build strategic program evaluation plans.
- Manage process to determine data necessary for program evaluation (e.g., collect and/or oversee collection of such data for all SOM educational programs).

Minimum Qualifications (Mandatory):

PHD in Psychometrics, Educational Assessment, Educational Measurement, Psychological Measurement, or a related field with an emphasis on statistics.

Preferred Qualifications:

Experience in assessing learners and evaluating programs in medical and biomedical education preferred.
Working knowledge of Learning and Assessment Management Systems (New Innovations, B-Line, Blackboard, and ExamSoft) and Qualtrics preferred.
Familiarity with medical education, curriculum development, and assessment standards associated with accrediting bodies such as the Middle States Commission on Higher Education (MSCHE), the Liaison Committee on Medical Education (LCME), the Accreditation Council for Graduate Medical Education (ACGME), and the National Institutes of Health (NIH), helpful.
Knowledge of medical and biomedical education programs, and familiarity with the academic medicine environment, preferred.

Special Knowledge, Skills, & Abilities:

Ability to work independently, collaboratively, and effectively with a diverse group of individuals that include students, learners, faculty, and staff in a dynamic and complex environment.
Knowledge of Classical Test Theory and IRT.
Training or experience in psychometric research including item response theory, scaling and equating methods.
Knowledge of and statistical analytic skills in measurement topics such as generalizability theory, Bayesian methodology, Rasch modeling, and various standard setting methodologies.
Proficient in quantitative and qualitative research methods.
Demonstrated knowledge of and skills in analytic and statistical modeling.
Demonstrated ability to independently assess and develop data management and statistical strategies to optimize reporting.
Demonstrated ability to analyze highly complex data sets.
Skills in data mining/management, data collection instruction, including ability to show data in a user-friendly format for a variety of audiences.
Demonstrated ability to prioritize tasks, manage time, and meet deadlines
Keen attention to detail.
Excellent writing and verbal communication skills.
Demonstrated ability to gather and secure data (both internally and externally) and write technical reports.
Proficiency with Qualtrics and Microsoft Office Suite (Word, Excel, PowerPoint, etc.)
Advanced skills in writing syntax/code for data manipulation.
Advanced skills using statistical software (e.g., SAS, SPSS, R, and STATA).
Knowledge of inferential statistical analyses, including independent and dependent samples t-test, chi-squared test, ANOVA, ANCOVA, Factor Analysis, simple and multiple linear regression, and simple and multiple logistic regression

Classified Title: Psychometrician
Working Title: Psychometrician & Program Evaluator
Role/Level/Range: ACRP/04/ME
Starting Salary Range: $60,945.00 - $83,865.00; Commensurate with experience
Employee Group: Full Time
Schedule: Monday - Friday, 8:30am - 5:00pm
Exempt Status: Exempt
Location: School of Medicine Campus, East Baltimore Campus
Department Name: Administration, Registrar General Administration
Personnel Area: School of Medicine

The successful candidate(s) for this position will be subject to a pre-employment background check.

If you are interested in applying for employment with The Johns Hopkins University and require special assistance or accommodation during any part of the pre-employment process, please contact the HR Business Services Office at jhurecruitment@jhu.edu. For TTY users, call via Maryland Relay or dial 711.

The following additional provisions may apply depending on which campus you will work. Your recruiter will advise accordingly.

During the Influenza ("the flu") season, as a condition of employment, The Johns Hopkins Institutions require all employees who provide ongoing services to patients or work in patient care or clinical care areas to have an annual influenza vaccination or possess an approved medical or religious exception. Failure to meet this requirement may result in termination of employment.

The pre-employment physical for positions in clinical areas, laboratories, working with research subjects, or involving community contact requires documentation of immune status against Rubella (German measles), Rubeola (Measles), Mumps, Varicella (chickenpox), Hepatitis B and documentation of having received the Tdap (Tetanus, diphtheria, pertussis) vaccination. This may include documentation of having two (2) MMR vaccines; two (2) Varicella vaccines; or antibody status to these diseases from laboratory testing. Blood tests for immunities to these diseases are ordinarily included in the pre-employment physical exam except for those employees who provide results of blood tests or immunization documentation from their own health care providers. Any vaccinations required for these diseases will be given at no cost in our Occupational Health office.

Equal Opportunity Employer
Note: Job Postings are updated daily and remain online until filled.

EEO is the Law
Learn more:
https://www.eeoc.gov/sites/default/files/migrated_files/employers/poster_screen_reader_optimized.pdf

To apply, visit https://jobs.jhu.edu/job/Baltimore-Psychometrician-&-Program-Evaluator-MD-21205/760150200/