Assessment in Health Professions Education The health professions, i.e., persons engaged in teaching, research, administra- tion, and/or testing of students and professionals in medicine, dentistry, nursing, pharmacy, and other allied health fields, have never had a comprehensive text devoted specifically to their assessment needs. Assessment in Health Professions Education is the first comprehensive text written specifically for this audience. It presents assessment fundamentals and their theoretical underpinnings, and covers specific assessment methods. Although scholarly and evidence-based, the book is accessible to non- specialists. • This is the first text to provide comprehensive coverage of assessment in the health professions. It can serve as a basic textbook in introductory and intermediate assessment and testing courses, and as a reference book for graduate students and professionals. • Although evidence-based, the writing is keyed to consumers of measure- ment topics and data rather than to specialists. Principles are presented at the intuitive level without statistical derivations. • Validity evidence is used as an organizing theme. It is presented early (Chapter 2) and referred to throughout. Steven M. Downing (PhD, Michigan State University) is Associate Professor of Medical Education at the University of Illinois at Chicago and is the Principal Consultant at Downing & Associates. Formerly he was Director of Health Programs and Deputy Vice President at the National Board of Medical Examiners and Senior Psychometrician at the American Board of Internal Medicine. Rachel Yudkowsky (MD, Northwestern University Medical School, MHPE, University of Illinois at Chicago) is Assistant Professor of Medical Education at the University of Illinois at Chicago. She has been director of the Dr. Allan L. and Mary L. Graham Clinical Performance Center since 2000, where she develops standardized patient and simulation-based programs for the instruction and assessment of students, residents, and staff. Assessment in Health Professions Education Edited by Steven M. Downing, PhD Rachel Yudkowsky, MD MHPE First published 2009 by Routledge 270 Madison Ave, New York, NY 10016 Simultaneously published in the UK by Routledge 2 Park Square, Milton Park, Abingdon, Oxon OX14 4RN Routledge is an imprint of the Taylor & Francis Group, an informa business © 2009 Taylor and Francis All rights reserved. No part of this book may be reprinted or reproduced or utilized in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identi fication and explanation without intent to infringe. Library of Congress Cataloging-in-Publication Data Assessment in health professions education / edited by Steven M. Downing, Rachel Yudkowsky. p. ; cm. Includes bibliographical references and index. 1. Medicine—Study and teaching. 2. Educational tests and measurements. I. Downing, Steven M. II. Yudkowsky, Rachel. [DNLM: 1. Health Occupations—education. 2. Educational Measurement— methods. W 18 A83443 2009] R834.5.A873 2009 610.71—dc22 2008042218 ISBN 10: 0–8058–6127–0 (hbk) ISBN 10: 0–8058–6128–9 (pbk) ISBN 10: 0–203–88013–7 (ebk) ISBN 13: 978–0–8058–6127–3 (hbk) ISBN 13: 978–0–8058–6128–0 (pbk) ISBN 13: 978–0–203–88013–5 (ebk) This edition published in the Taylor & Francis e-Library, 2009. To purchase your own copy of this or any of Taylor & Francis or Routledge’s collection of thouasands of eBooks please go to www.eBookstore.tandf.co.uk. ISBN 0-203-88013-7 Master e-book ISBN To my husband Moshe Looking forward to the next billion seconds and more To our children Eliezer and Channah Who bring us much pride and joy And in memory of our son Yehuda Nattan May his memory be a blessing to all who loved him ~ RY Contents L IST OF F IGURES ix L IST OF T ABLES xi P REFACE xiii Steven M. Downing and Rachel Yudkowsky C HAPTER-SPECIFIC A CKNOWLEDGMENTS xix 1I NTRODUCTION TO A SSESSMENT IN THE H EALTH P ROFESSIONS 1 Steven M. Downing and Rachel Yudkowsky 2VALIDITY AND I TS T HREATS 21 Steven M. Downing and Thomas M. Haladyna 3RELIABILITY 57 Rick D. Axelson and Clarence D. Kreiter 4GENERALIZABILITY T HEORY 75 Clarence D. Kreiter 5STATISTICS OF T ESTING 93 Steven M. Downing 6STANDARD S ETTING 119 Rachel Yudkowsky, Steven M. Downing, and Ara Tekian 7WRITTEN T ESTS: C ONSTRUCTED-R ESPONSE AND S ELECTED-R ESPONSE FORMATS 149 Steven M. Downing 8OBSERVATIONAL A SSESSMENT 185 William C. McGaghie, John Butter, and Marsha Kaye 9PERFORMANCE T ESTS 217 Rachel Yudkowsky vii 10 SIMULATIONS IN A SSESSMENT 245 William C. McGaghie and S. Barry Issenberg 11 ORAL E XAMINATIONS 269 Ara Tekian and Rachel Yudkowsky 12 ASSESSMENT P ORTFOLIOS 287 Ara Tekian and Rachel Yudkowsky L IST OF C ONTRIBUTORS 305 I NDEX 311 CONTENTSviii List of Figures 1.1 George Miller’s Pyramid 4 4.1 G Coefficient for Various Numbers of Raters and Stations 86 6.1 Eight Steps for Standard Setting 122 6.2 Hofstee Method 139 6.3 Hofstee Example 141 6.4 Borderline Group Method 143 6.5 Contrasting Groups 144 6.6 Minimizing Passing Errors 144 8.1 Comparison of the Miller Pyramid and Kirkpatrick Criteria for Learning Outcomes 191 9.1 Miller’s Pyramid: Competencies to Assess with Performance Tests 218 9.2 Essential Elements of a Standardized Patient Case 222 9.3 Rating Scale Anchors 225 9.4 Sample Rubric for Scoring a Student’s Patient Chart Note 226 9.5 An 8-Station OSCE for an Internal Medicine Clerkship 228 9.6 OSCE Case Example 231 9.7 Construct Underrepresentation 234 9.8 Creating Blueprint Specifications for an OSCE 235 9.9 Some Frameworks to Assist in Blueprinting OSCEs 236 11.1 Miller’s Pyramid: Competencies to Assess with Oral Exams 270 12.1 Kolb’s Experiential Learning Cycle 288 12.2 Miller’s Pyramid: Sample Elements that can be Included in an Omnibus Assessment Portfolio 290 12.3 Assessment Portfolio Steps 292 ix [...]... the health professions Such books lack essential topics of critical interest to health professions educators and may contain many chapters that are of little or no interest to those engaged in education in the health professions Assessment in Health Professions Education presents chapters on the fundamentals of testing and assessment together with some of their theoretical and research underpinnings... support its validity (Downing, 2002; Welch, 2006) Chapter 7 discusses both constructed and selected response written tests Observational of Clinical Performance Assessment of clinical performance during clinical training is a very common form of assessment in health professions education These types of assessment range from informal observations of students in clinical settings to very formal (and sometimes... methodology to health professions education In Chapter 5, Downing presents some basic information on the statistics of testing, discussing the fundamental score unit, standard scores, item analysis, and some information and examples of practical hand-calculator formulas used to evaluate test and assessment data in typical health professions education settings Standard setting or the establishment of passing scores... scope of this book, which presents current information about both assessment theory and its practice in health professions education The focus of this book is on the assessment of learning and skill acquisition in people, with a strong emphasis on broadly defined achievement testing, using a variety of methods Health professions education is a specialized discipline comprised of many different types of... provides the reader with guidelines for performance assessment using techniques such as standardized patients and Objective Structured Clinical Exams (OSCEs) These methods are extremely useful in skills testing, which is generally a major objective of clinical education and training at all levels of health professions education xvi PREFACE High-tech simulations used in assessment are the focus of Chapter... This is not possible in educational measurement, obviously, since even the least capable student will have some minimal knowledge (True ratio scales are often found in the physical sciences, but not in the social sciences.) The main point of this discussion of number-types is that most of the assessment data we obtain in health professions education is considered or assumed to be interval data, so that... of data gathering from multiple raters about the performance of health professions students in actual clinical settings, with real patients over lengthy periods of time Typically, many of these observational assessment methods rely on checklists and rating forms, completed by faculty and other instructors in clinical settings Many of these observational assessments carry major weight in overall or... “competence” in health professions education Both the Competencies and the Toolbox are sufficiently general to be useful in many different areas of health professions education, and at all levels of training We recommend that you download these documents and become familiar with their content Table 1.1 summarizes the thirteen assessment methods included in the Toolbox Many of the methods noted in the Toolbox... scale, but is rarely if ever possible to obtain in educational measurement or the INTRODUCTION TO ASSESSMENT 13 social sciences A true ratio scale has a meaningful zero point, so that zero means “nothingness.” This mean that if we could devise a legitimate ratio testing instrument for measuring the achievement of nursing students in biochemistry, students scoring 0 would have absolutely no knowledge of... in various forms are used widely in health professions education worldwide This chapter provides information on the fundamental strengths and limitations of the oral exam, plus some suggestions for improving oral exam methods Portfolio assessment, discussed in Chapter 12, is both old and new This method is currently enjoying a resurgence in popularity and is widely applied in all levels of health professions . contain many chapters that are of little or no interest to those engaged in education in the health professions. Assessment in Health Professions Education. without intent to infringe. Library of Congress Cataloging -in- Publication Data Assessment in health professions education / edited by Steven M. Downing, Rachel