Developing a Strategy for Using Technology-Enhanced Items

What is your strategy for incorporating technology-enhanced items into your assessment program?

Technology-enhanced items (TEIs) are computer-/tablet-delivered assessment items that take advantage of computer technology to present test takers with innovative, interactive situations and capture their responses in ways not possible with traditional multiple-choice and constructed-response items. TEI item types are many and varied. Examples of response formats include graphing; completing complex math and science tasks; dragging and dropping text or images; matching, ordering, and sorting texts and concepts; filling in the blanks; identifying/highlighting keywords/phrases in reading selections; graphing (e.g., bar graphs, line graphs, histograms, dot plots); clicking on images (hotspots), buttons, numbers, or text; and completing audio (voice) or video tasks. The use of animation and game-based items/tasks has brought a new dimension to TEIs. Following are three examples.

Example 1: (Hot Spot)

In a hot spot question, using their cursor, students identify a location on an image that represents the correct answer to the question.

Example 2: (Matched Pairs)

When presented with this format, students drag answers from one section to another to create matching pairs. As shown in the image below, students will drag four answers into the boxes in order to answer the question. In some cases, not all answers will be used. In the example below, all answers are used.

Example 3: (Click on Buttons) Broken Calculator

Presented with this task, students click on the available calculator buttons to obtain a result of 19 on the screen. (Earlier attempts are also displayed on-screen.)

Please follow these links to view more involved items/animations provided by Vretta Inc.:

• Geometry Work Space Demo
• Room Remodeling Demo
• Chill Drink Demo
• Animal Rescue Demo

In recent years, as large-scale assessments have moved from paper- to computer-based modes of delivery, test developers have enthusiastically embraced technology-enhanced items (TEIs), as they can provide advantages over traditional item types. Examples of potential advantages of well-designed TEIs include the following:

• Present test takers with real-life scenarios and consequently more authentic assessments
• Provide more engaging testing experiences
• Assess higher-level thinking skills, problem solving, and other cross-curricular competencies
• Provide information about thought processes (how test takers formulated their responses) via log-file data (e.g., time spent on items, number of clicks)
• Offer the possibility of providing diagnostic information
• Allow for efficient computer scoring

There are, however, some possible downsides to consider. For instance, generally, they are more expensive to develop; more involved items require advanced computer-programming expertise; and their psychometric properties are not always as well understood compared with traditional item types.

Jumping on the TEI bandwagon, simply because these item types are novel and exciting, can lead to serious assessment issues. It is important to note that, like paper-based assessments, decisions about the choice of item formats should be driven by the content/construct(s) to be measured. Careful thought must be given to if/how TEIs can positively contribute to an assessment. Following are some guiding questions that may be useful in developing a coherent strategy for incorporating TEIs into large-scale assessments:

• How can we decide which TEI formats are appropriate to measure performance associated with given curriculum learning expectations/objectives?
• How can we ensure TEIs are well-designed?
• How can we measure and demonstrate construct validity for individual items, sets of items, and the assessment overall? (What is the body of evidence that will give confidence the assessment has construct validity?)
• How can we meaningfully incorporate TEIs into the design of our assessment?

RMJ Assessment has conducted a great deal of research in the field of e-assessment and can offer considerations and advice toward designing and developing plans for incorporating TEIs into large-scale assessments. Should you wish to learn more about the services we offer, please contact us by e-mail at richard.jones@rmjassessment.com or call (647) 234-5346.