MULTIPLE CHOICE QUESTIONS DESIGN AND EVALUATION Helen Huntley Senior English Language Fellow Ministry of Education and Training Hanoi, Vietnam Workshop Goals    To review the advantages and disadvantages of multiple choice tests To explore tips and techniques for writing successful multiple choice items To evaluate and correct multiple choice questions and answers Objective Testing “Tests in which the learning material is divided into small units, each of which can be assessed by means of a test item with a single correct answer that can be specified in advance.” Press Reed, J (2000) Assessing vocabulary Cambridge University Advantages of Multiple Choice Tests         good discrimination among levels of ability high reliability valid indicators of overall language ability highly practical: convenient to administer well-established procedures to analyze items writing ability does not affect performance cost-effective format - easy scoring and grading familiarity of students with MCQs Disadvantages of Multiple Choice Tests            tendency to treat words as independent linguistic units difficulty in construction field-testing, analysis, refinement required 25% chance of guessing correctly items may test knowledge of distractors wrong answer may result from lack of understanding of syntax (English) permits only limited sampling of total knowledge tests only recognition knowledge restricts what can be tested washback may be harmful cheating may be facilitated Terminology   Every multiple choice item has a stem, which presents a stimulus and several options or alternatives to choose from One of these options, the key, is the correct response The others serve as distractors Match the Test Items to the Blueprint and Task Specifications          Addresses curriculum objectives in terms of performance (can do) Outline of the test Skills to be included Item types and tasks Topics to be covered Elicitation and response formats for items Number of items in each section Time allocation Plan for scoring Using Item Indices to Accept, Discard, or Revise Items    Item facility: the extent to which an item is easy or difficult for the proposed group of test takers Item discrimination: the extent to which an item differentiates between high- and lowability test-takers Distractor efficiency: the extent to which the distractors ‘lure’ a sufficient number of test takers Multiple Choice Tips I         Each test item should be designed to measure a specific objective The question or task should be clear, testing only one concept The answer should involve interpretation of the material, not just background knowledge A context should be provided for the items The items should not show an insensitive or unfair bias The number of response options should be the same in each question Ambiguity in the answers must be avoided; the intended answer should clearly be the only correct one The options should all be similar in length and level of difficulty Multiple Choice Tips II         Items at different levels of difficulty should be included: start with the easiest first Both the stem and options should be stated as simply and directly as possible Writing illogical or nonsensical distractors should be avoided Extraneous clues should not influence the answer The correct answers should appear in all positions (A, B, C, D) Words should not be repeated in the answers A negative ‘stem’ should be avoided The questions should not be dependent on previous items Appropriate punctuation should be used 10 MCQ#11 Although the plan has some disadvantages, they are outweighed by the _ a b c d advantages negatives drawbacks problem 23 MCQ#12 Roberto is lazy He always takes a siesta _ the afternoon a b c d in during at throughout 24 MCQ#13 My eyesight has really been deteriorating lately I wonder if I need glasses I think I’d better go to the _ to have my eyes checked a pediatrician b dermatologist c optometrist d audiologist 25 MCQ#14 We went to visit the pagodas, _ fascinating a b c d Which Which Which Which were were were were beautiful especially holy special 26 MCQ#15 Where did Jane go after the birthday party last night? a b c d Yes, she did Because she was tired To her friend’s house She went home around ten o’clock 27 MCQ#16 Where did George go after the party last night? a b c d Yes, he did Because he was tired To Elaine’s place for another party Around eleven o’clock 28 Your Turn Complete the worksheet to find problems in multiple choice question development 29 Multiple Choice Editing Questions/Error Recognition The abrasively action of the wind wears A B away softer layers of rock C D 30 Multiple Choice Editing Questions/Error Recognition Researchers have discovered that the A application of bright light can sometimes B be uses to overcome jetlag C D 31 Multiple Choice Editing Questions: Paragraph Ever since supermarkets first appeared, they have been take over the world Supermarkets have changed people’s lifestyles, yet and at the same time, changes in people’s lifestyles have encouraged the opening of supermarkets As a result this, many small stores have been forced out of business Moreover, some small stores will be able to survive this unfavorable situation 32 GapFill/Cloze Multiple Choice In years to come, zoos will not (1) _ be places where animals are exhibited (2) _ the public, but repositories where rare species can be saved from extinction (3) _ captive breeding… a also b only c even d ever a by b near c to d into 33 Vocabulary Multiple Choice I disagree with a few of his opinions, but _ we agree a deliberately b conclusively c essentially d immensely 34 Vocabulary Multiple Choice Because fewer people are taking expensive vacations, the tourist industry is in a _ a choke b grope c grumble d slump 35 Item Writing Practice Further practice in designing multiple choice test items will be provided in specific language skill assessment sessions 36 Any Questions? ? b ? c ? d ? a 37 ... advantages and disadvantages of multiple choice tests To explore tips and techniques for writing successful multiple choice items To evaluate and correct multiple choice questions and answers... problems in multiple choice question development 29 Multiple Choice Editing Questions/ Error Recognition The abrasively action of the wind wears A B away softer layers of rock C D 30 Multiple Choice. .. Readability Grade (MSW) 11 Sample MC Questions Evaluate the following multiple choice items    What is the objective of each question? Identify problem areas (see Multiple Choice Tips) Correct/improve