| More on “All
That Apply” Questions
by Steven B. Just
Two weeks ago I wrote an article
on "All That Apply" questions
discussing the fact that many test creators don’t use
them because learners find them “difficult” (which
they are, compared to traditional multiple choice questions).
This raises the question:” As compared to other question
types, how difficult are they?” Let’s see if we
can answer that question using some simple math.
First of all we have to control for content
difficulty and assume that the questions we are comparing
all have the same degree of content difficulty even though
they are of different type. For example, though All That Apply
questions are generally more difficult than multiple choice
questions, it is possible to write a very difficult multiple
choice question and a very easy All That Apply question about
the same topic (US presidents):
Multiple Choice Question:
Who was the 21st president of the United
States?
Rutherford B. Hayes
James Garfield
Grover Cleveland
Chester A. Arthur
Ulysses S. Grant
Unless you know a lot more about US History
than I do, that’s a hard question.
All That Apply Question:
Select the presidents who served BEFORE
Abraham Lincoln
George Washington
Bill Clinton
Thomas Jefferson
John Kennedy
George Bush
And that’s an easy question.
By extension, this demonstrates that it is possible
to write multiple choice questions and All That Apply questions
of comparable content difficulty. In other words, we are trying
to make an “all things being equal” comparison.
Assuming that there is no difference in content
difficulty let’s look at three question types in order
of their “intrinsic” difficulty:
1. True/False
2. Multiple Choice
3. All That Apply
(You’ll see why I introduced True/False
in a moment.)
Let’s start with multiple choice questions.
Most of the tests I see tend to have an average score of around
90%. So, if this is the case, and if all the questions are
multiple choice then the average difficulty of any one multiple
choice questions is .9, meaning, on average, 90% of the test
takers will get that question correct.
Now let’s look at True/False questions.
Clearly (all things being equal) they are easier than multiple
choice questions. For a content-equivalent test of T/F questions
rather than five-choice multiple choice questions the average
score would now be 94% (read my article
to see why). Thus the difficulty level of each T/F question
is .94. (Difficulty values go in the opposite direction of
intuitive. A difficulty of 1 is easiest and a difficulty of
0 is hardest.)
Now let’s look at the All That Apply question
difficulty. Again, let’s assume the same content difficulty.
Consider the question above:
Select the presidents who served BEFORE
Abraham Lincoln
George Washington
Bill Clinton
Thomas Jefferson
John Kennedy
George Bush
If you think about the typical All That Apply
question it is really a series of True/False Questions, where
each choice may be true or false. So in effect this question
could be rewritten as five true/false questions:
True or False: George Washington served BEFORE
Abraham Lincoln
True or False: Bill Clinton served BEFORE Abraham Lincoln
etc.
Using our difficulty assumptions from above,
if this single question were presented as five separate True/False
questions, each would have a difficulty of .94 and the average
test score would be 94%. So, then why is an All That Apply
question so much more difficult? Because, unless you give
partial credit the learner must get ALL of the parts (individual
T/F questions) correct to earn a point. Even one incorrect
answer results in all of the answers being marked incorrect.
Now let’s see if we can quantify the “intrinsic
difficulty” of an All That Apply question (again, without
partial credit allowed). If the probability of getting one
T/F question correct is .94 then the probability of getting
all five correct is .94x.94x.94x.94x.94 = .73.
So a test consisting of only five-choice
All that Apply questions with no partial credit and
the same content difficulty as the multiple choice and true/false
examples we used above will have an average score of 73% as
compared to 90% on the multiple choice test and 94% on the
True/False test. Or, put another way, a test consisting of
only five-choice All That Apply questions, without partial
credit, will have an average score 17% less than a comparable
test consisting of only multiple choice questions!
(Oh, Chester A. Arthur was the 21st president
of the US.)
|
