News and events

“Item Writing Guidelines: Do We Agree?”
by Steven B. Just

In my workshop on Best Practices in Test Validation I spend some time talking about guidelines for question creation, since a test can’t be valid unless the questions themselves are valid. I have eight multiple-choice question writing guidelines that I discuss:

• One piece of information per question
• All information should appear in stem
• All choices should be parallel in form
• Do not use double negatives
• Avoid “all of the above”
• Arrange responses in logical order
• Correct choice and distractors should be of the same length
• All choices should be in syntactic and semantic agreement with the stem

I am aware that there are other guidelines, some of which make sense to me:

When a question calls for a personal pronoun, write questions in the second person, never in the impersonal third person:

Bad Question: One should always …

Good Question: You should always…

and others with which I disagree:

Do not use “None of the above” as a choice.

All this begs the questions: How many question-writing guidelines are there? Are they all correct? Who makes up the guidelines anyway?

The answers to these three questions are, respectively: 31. No. Anyone can.

The answer to the third question implies the answer to the second. Since there is no central authority adjudicating question-writing guidelines, just about anyone is free to create his or her own guidelines. Thus many “guidelines” are not guidelines per se, but one person’s opinions.

Now for the first answer: How do I know that there are 31 guidelines? Because in a paper I came across recently “A Review of Multiple-Choice Item-Writing Guidelines for Classroom Assessment” Applied Measurement in Education, I5(3) 309-334, the authors Thomas Haladyna, Steven Dowling and Michael Rodriguez reviewed 27 textbooks and 27 research studies and compiled all the multiple-choice question writing guidelines they could find.

They then tabulated in what percentage of the references each guideline was:

• Cited and supported
• Cited and not supported
• Not cited

They reasoned, with some justification I believe, that those guidelines that are cited AND supported in the vast majority of studies are valid guidelines.

Let’s look at some of their results:

• The vast majority of the guidelines (24), if they were cited, were always supported, which implies that there is a strong consensus on at least 24 item-writing guidelines.
• The only guideline that was always cited and always supported was: “Keep the central idea of the question in the stem.”
• 48% of the references believe “None of the Above” should not be used. 44% said it could be used, but sparingly.
• Avoid “All of the Above,” to me one of the cardinal guidelines, was actually opposed by 22% of the references.
• Make distractors plausible was cited positively by 96% of the references. 4% didn’t cite it at all. No one opposed it.
• Many obvious guidelines were uncited in a substantial percentage of the references. I imagine the authors thought these guidelines were so obvious they weren’t worth mentioning. So “Edit and proof”,” Use correct grammar”, and “Use humor sparingly” were uncited in 67%, 48% and 85% of the studies, respectively.
• 15% were against “using humor sparingly.” Did this mean they were opposed to using humor at all or they thought humor should be used more often? The answer according to the authors is the latter, though they do warn against humor in high stakes testing.

My next article will discuss some other guidelines and the authors’ findings in some more detail, some of which are rather counterintuitive.