How Are Your Item-Writing Skills?

Too often, assessment items (questions and tasks) are poorly written, causing frustration and anxiety for test-takers and casting doubt on the validity and accuracy of assessment results. Many senior practitioners have commented that the crafting of high-quality student assessment items takes a great deal of skill and experience. The writing of high-quality assessment items involves both science and art. The science (psychometric analysis and research) provides us with data/information that informs the guidelines for effective item writing. Educators and professional item writers bring their artistry to the wording of items to give students the opportunity to demonstrate their best performances.

Clearly, the information below is not meant to be comprehensive; however, attention to these basic guidelines will contribute to the creation of quality assessment items.

All Item Types

In general, across all item types (broadly categorized as objective, constructed-response, and performance), quality items should:

• assess important points/information.
• reflect the intent of the learning objectives/expectations and the test blueprint or specifications.
• present a clear, specific task.
• be fair and unbiased (avoid potential sources of bias, including gender, cultural and geographic references, unfamiliar contexts, item formats preferred/easier for given groups as opposed to others).
• avoid the use of tricks, as well as unnecessary/extraneous and misleading information.
• avoid the use of absolutes (e.g., always, never, none, all).
• be independent (one question should not provide cues/clues to another).
• be written in clear, straightforward, familiar language (at the appropriate reading level) using good grammar, punctuation, and spelling.

Objective Items (e.g., multiple-choice, matching, true-false, completion)

In addition to the aforementioned guidelines (using multiple-choice questions as an example), quality items should:

• present a complete question or task in the stem that could be answered without reference to the response options (question format is more natural than incomplete statement; however, if blanks are used, they should come at the end).
• use positive phrasing in the stem (use negatives sparingly; avoid double-negatives).
• use alternatives such as “all of the above” and “none of the above” sparingly.
• emphasize key words (e.g., most, least, only, best) by underlining, using bold face type, or capitalizing all letters.
• use at least four response options.
• contain one correct or best response.
• have mutually exclusive response alternatives.
• have response options that are grammatically consistent with the stem.
• use distractors which represent common errors and misconceptions.
• provide plausible distractors (throw-away responses should be avoided).
• avoid the use of qualifiers (e.g., sometimes, occasionally, usually, perhaps).
• avoid repeating words in the response options that appear in the stem.
• have response options arranged in a logical sequence (e.g., shortest to longest) to enhance readability.
• have response options that contain grammatically parallel language (e.g., content, length, technical level, and amount of specificity).
• randomly distribute the location of the correct response throughout the assessment, having approximately the same proportion of alternatives being the correct response.
• be able to be completed in one to two minutes.

Constructed-Response Items/Tasks (short answers and longer written responses)

In addition to the guidelines applicable to all item types, constructed-response items and their instructions should:

• measure learning objectives/expectations that can’t readily be assessed via objective items (require higher-level thinking).
• contain sufficient information such that the examinee knows all of the response requirements.
• elicit a range of student performance on the scoring rubric used (particularly for longer open responses).
• be developed prior to creating objective items for reading selections (if objective items are written first, it may be difficult to create meaningful constructed-response items).
• be able to be completed in four to six minutes on average (for short answers) and 20 minutes or more (for longer constructed responses).

Performance Test Items/Tasks (e.g., laboratory experiments, artistic displays, written/oral presentations)

In addition to the guidelines applicable to all item types, performance assessment items/tasks and their instructions should:

• be constructed to measure specific performance(s)/behaviour(s) based on the targeted learning objective(s)/expectation(s).
• provide the test-taker with all relevant information including a clear explanation of the assignment/task, the performance expectations, and how the quality of the performance will be evaluated.
• be as authentic (life-like) as possible.

Summary

Guidelines, such as those presented above, are useful for developing and evaluating assessment items; however, checklists are not the only answer: there is no substitute for actual item writing experience. Many testing organizations hire practising educators to draft items, which often need to be “professionalized” by in-house experts. Using teachers to draft items is a positive approach, because they bring fresh, authentic ideas from the field, and subsequently, take their training and experience back to their schools and classrooms. The downside is that oftentimes items are unusable or require significant reworking, which can be costly to the organization. As a consequence, some testing organizations rely on highly-qualified and experienced freelance item writers to author assessment items.

To reiterate, quality item writing combines elements of both art and science.

______________________

QUIZ

Can you identify the flaws in the items below?

1. The dog in the story is

A    walking to the park

B    walking to the forest

C    walking to visit a friend

D    walking to the town

 

2. Which of the following is not red?

A    strawberries

B    pears

C    apples

D    tomatoes

 

3. In the story, Sarah’s aunt tells him the age of the tree. How old does she tell her the tree is?

A    one hundred years

B    less than twenty-five years

C    more than fifty years

D    more than ninety years

 

4. In national elections in Canada, the Prime Minister is chosen by

A    the people

B    the House of Commons of Parliament

C    the winning political party

D    the Senate

 

5. Which of the following is a synonym of the word “astonishing”?

A    irreproachable

B    arrogant

C    sociable

D    astounding

 

6. Compost is a mixture of

A    a variety of soils

B    kitchen waste

C    tins and bottles

D    black soil, leaves, grass, and vegetable scraps

 

7. A land mass surrounded by water is called an

A    isthmus

B    continent

C    island

D    peninsula

 

8. RMJ Food Mart has reduced prices by 11% on everything in the store as part of an annual promotion. If a Strawberry Swirl™ ice cream cone cost $2.99 before the price reduction, approximately how much does it cost now?

A    $2.78

B    $2.66

C    $2.60

D    $2.72

 

9. Why do you think the young boy in the story is sad? Use information from the story to support your answer.

 

10. Who did Ahmad go to visit in the story?

 

11. Compare the writings of Charles Dickens and Mark Twain with regard to setting, character development, and writing styles.

 

12. How do you think the girl in the story feels? Use details from the story to support your answer.

 

13. This performance task is meant to help students appreciate the diversity of people, places,    and things that comprise communities. At the end of an instructional unit, elementary-school  students are tasked with the following:

 On a large sheet of Bristol board, draw a picture of a community (either real or imaginary).

• On your map, label people, buildings, and natural features (forests, open spaces, waterways, animals).
• Write a description of what people do, how they use the different spaces in their community, and what makes their community inviting or uninviting to live in.
• Present your work to the class.

______________________

Further information about item writing test development can be found in my book: Large-Scale Assessment Issues and Practices: An Introductory Handbook (2014), which can be accessed at https://www.amazon.ae/Large-Scale-Assessment-Issues-Practices-Introductory/dp/0968485731.

RMJ Assessment provides a wide range of assessment services, which are outlined on our website: https://www.rmjassessment.com/large-scale-assessment/.