News and events

“Item Writing Guidelines: Do We Agree?” (Part 2)
by Steven B. Just

In Part 1 of this article I reviewed item writing guidelines from the article: “A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment” Applied Measurement in Education, 2002, 15(3) 309-334, by Thomas Haladyna, Steven Dowling and Michael Rodriguez. In this continuation I would like to look a little more closely at two of the 31 item-writing guidelines they include in their taxonomy:

Base each item on important content to learn; avoid trivial content.

Recently I did a test validation project for the sales training department of a major pharmaceutical company. As part of the validation process we gathered together three subject matter experts (subject trainers from their sales training department) to review each of the questions (which had been written by one of their vendors). We reviewed each question to make sure it tested its respective learning objective, to ensure that the correct answer was indeed correct, to make sure all the distractors were plausible and to ensure that each question met recognized standards of valid question writing. What surprised me somewhat (I’ve done many test validations and this was the first time this ever happened) was that the trainers threw out about 10% of the questions, not because there was anything inherently wrong with these questions, but because in their view, the questions were testing information that, while in the learning system, was not relevant to the job. Or, as the sales trainers commonly expressed it: “They would never need to know that when talking to a doctor. That would never come up.” Which raises the question: “How do these “irrelevant” questions get into the item pool?”

Consider who writes questions. (What I’m going to discuss may be somewhat peculiar to the domain of pharmaceutical sales training.) Questions are traditionally written by vendors’ content experts (the same folks who write the learning systems). But who are these people? Typically they are medical writers whose backgrounds are in biology, medicine or pharmacology. I would venture that none of them have ever actually been pharmaceutical sales representatives. These two populations (medical writers and pharma reps) just don’t tend to overlap. So, while medical writers are subject matter experts, they tend not to know what information is actually important in day-to-day rep/doctor interactions.

So, if you are in a pharmaceutical sales training department, the bottom line is: question validity also includes job relevancy.

Use novel material to test higher level learning. Paraphrase textbook language or language used during instruction when used in a test item to avoid testing for simple recall.

This is a perennial problem. In the item pools I review for validity it is often the case that 100% (or close to that) of the questions are written at the level of simple recall. In fact most of the questions are not only recall but the stem and correct answer are literally lifted verbatim from the text. I don’t want to minimize the difficulty of writing questions for higher level thinking skills. It’s hard—and time consuming. But this guideline does provide a good method for at least raising the question above pure memorization: Paraphrase the learning system text in the question. Do not lift it verbatim.