News and events

A Critical Look at Learning Objectives and Assessments
By Steven B. Just

Valid criterion-referenced testing theory instructs us to closely tie test questions to learning objectives (Criterion Referenced Test Development, Shrock and Coscarelli, 2007). There is unassailable logic here: The learning objectives define the scope of what the student is to learn and the test questions ensure the acquisition of this knowledge.

But, as the saying goes: “In theory there is no difference between theory and practice, but in practice there is,” and I have growing reservations about this practice. Apparently at least one other person also feels this way. Will Thalheimer, in his very positive review of the latest edition of the Shrock and Coscarelli book, states in his blog (www.willatworklearning.com) :

They use the methodology of relying on a single objective to guide the process of both instructional design and evaluation. I am now advocating to free instructional-design objectives from the crazy constraint of being super-glued to the evaluation objectives.

Let me first say that I am all for using learning objectives (albeit with caveats) and I understand the need to tie evaluation back to instruction, but let me point out just a few practical problems that result from a literalist adherence to this methodology.

Which Learning Objectives Do We Test On?

Let’s look at a typical (print-based or on-line) learning system. (I am most familiar with the learning systems used in the pharmaceutical industry so I’ll use these as an example.) In the pharmaceutical industry, learning systems generally consist of a number of modules (typically five to seven). Each module is further divided into lessons or sections (five to ten, sometimes more). Each lesson then begins with its requisite learning objectives (usually around five to ten). So, in the learning system as a whole there are generally anywhere between 125 and 700 distinct learning objectives.

For formative evaluations at the module level it is sometimes practical to test on every learning objective — as the theory tells us we are supposed to — but what do we do for the final summative evaluation? Short of giving a 300 question test or giving multiple summative exams (as many as seven if you do the math – see below) the only practical solution is to test on some learning objectives and not others. In effect, we are forced to conclude that some learning objectives are more important than others — some become nice-to-knows and others become need-to-knows. This is not always palatable but it’s the only practical course of action.

How Many Questions Does It Take to Test a Learning Objective?

For practical reasons we usually like to give tests that are no longer than an hour or so. If we give the students 60 to 90 seconds per question, the test can then only have 40 to 60 questions — let’s say 50, which, in my experience, is a typical summative exam length. So that means we’ve somehow whittled our 300 learning objectives down to the 50 that we are going to test on. This means that we have excluded 250 learning objectives from our summative evaluation. That’s an awful lot of nice-to-knows! Not ideal, but let’s assume we’ve done that. We now have 50 learning objectives and 50 questions, or one question per learning objective. Two obvious problems present themselves:

1. Is one question enough to test a learning objective? The answer here is sometimes yes, often no. And even if one question “covers” the objective, is one question enough to ensure — taking guessing and the standard error of measurement into account – that the student has truly “mastered” the objective? Doubtful.
2. If we adhere to the notion that each need-to-know learning objective is independently critical knowledge that must be mastered this implies that the passing score on the test must be 100%! Why? Any score less than 100% means that the student has not mastered at least one element of need-to-know knowledge.

Can We Please Test Above the Knowledge/Recall Level?

We all know that most tests are written at the Knowledge/Recall level of Bloom’s Taxonomy. And we all know why: It’s hard and time consuming to write higher-level questions. And there’s another reason: Learning objectives let us off the hook.

How are learning objectives our Recall enablers? The logic goes like this:

1. We know from valid-testing theory that questions need to be written to learning objectives.
2. What level of Bloom’s are most learning objectives written at? Knowledge, of course: “You will be able to state…” “You will be able to list…” You will be able to name…” “You will be able to define…” (The learning objectives that I see are usually very narrowly written.)
3. So, we follow valid-testing theory and write the corresponding questions to the narrow learning objectives – at the Knowledge/Recall level.
4. We’re just doing what theory tells us to do.

What is To Be Done?

Thalheimer has the right idea: Let’s decouple learning objectives from evaluation objectives. Writing independent evaluation objectives provides several benefits:

1. It will permit us to write fewer, critical, broader objectives on which to test.
2. Because there will be fewer, broader objectives we can actually have several — or sometimes many — questions per objective, permitting a student to miss a question or two on an objective without failing to master the entire objective. Or, put another way, we can have a legitimate passing score of less than 100%.
3. We can take this re-writing opportunity to write the evaluation objectives — and thus their corresponding questions — at higher Bloom’s levels.

Writing a separate set of evaluation objectives adds a step to the process (i.e. it’s more work) but it does address the practical problems that arise from the traditional learning objective-based methods.