Introduction
In Canada, there is no federal ministry of education. This means that each of the ten provinces and three territories establishes its own policies related to curriculum and instruction, assessment and evaluation, and reporting of student learning progress. School boards set their own local policies within policy frameworks established by the provincial/territorial ministries of education.
The Education Quality and Accountability Office (EQAO), an arms-length agency of the Ontario Ministry of Education, has been developing and administering large-scale student assessments in the province for more than 25 years. Beginning with the 2021-2022 school year, EQAO’s assessments have been delivered digitally.
The provincial assessments are meant to measure students’ achievement at key stages of their learning according to the expectations of The Ontario Curriculum. The assessments include the Primary Division (Grade 3) and Junior Division (Grade 6) Assessments of Reading, Writing and Mathematics that test the skills students are expected to have acquired by the end of Grade 3 and Grade 6, respectively; the Grade 9 Assessment of Mathematics that assesses the math skills students are expected to have acquired by the end of Grade 9; and the Ontario Secondary School Literacy Test (OSSLT) that measures whether students are meeting the minimum standard for literacy (reading and writing) across all subjects to the end of Grade 9.
The purpose of this article is to provide a brief overview of the EQAO’s provincial e-assessments and describe some important benefits that EQAO and the province have realized as a result of the transition to digital assessments.
Overview of Provincial Assessments
Primary and Junior Assessments of Reading, Writing and Mathematics[1]
A total of approximately 140,000 students (English and French combined) in each of Grade 3 and Grade 6 are assessed annually on the reading, writing and mathematics skills defined in The Ontario Curriculum, Grades 1-8: Language (2023) and The Ontario Curriculum, Grades 1-8: Mathematics (2020). The Assessment blueprints can be accessed in the associated framework documents (sources referenced above). (Please note that revised frameworks will be available by the end of January, 2024 and may be accessed via the EQAO website.) The assessment is made up of three components: an introductory session in which students have an opportunity to experience a sample test that demonstrates the kinds of items (questions) that will appear on the actual assessment, as well as the available online tools such as text-to-speech, zoom in/out, high contrast, highlighter, eraser and calculator. The second component is the actual assessment. The language (reading and writing) assessment is made up of four sessions, with each session designed to be completed in 25 to 35 minutes. The mathematics assessment also contains four sessions, with each designed to be completed in approximately 30 minutes. Students complete each of the four sessions in one sitting. Sessions can be taken back-to-back with breaks in between or on different dates and times.
The language component of the Primary- and Junior-Division assessments consists of a variety of question types, including selected-response (e.g., drag-and-drop, drop-down menu, checklist, single- and multi-selection) and constructed-response items. The mathematics component consists entirely of selected-response questions. For all EQAO assessments, the vast majority of items students respond to are termed “operational,” meaning they count toward students’ final results; a relatively small number of field-test items, which do not count toward their outcomes, are also presented to students. (For all EQAO assessments, fewer than 10% of the total items on the test are used for field-testing purposes.) Grade 3 and Grade 6 sample tests can be viewed via the website at: eqao.com/the-assessments/primary-division and eqao.com/the-assessments/junior-division, respectively.
The language component of the primary and junior assessments uses a linear test design, in which several equivalent test forms of fixed length are assembled according to the same content and statistical specifications prior to administration of the assessment. The mathematics component employs a multi-stage computer adaptive testing model (msCAT), in which blocks of assessment items (designed to represent various difficulty levels) are presented to students based on their ongoing performance on the test. After the operational items are scored, EQAO uses Item Response Theory (IRT) to calculate a scaled/standardized score that takes into account the student’s responses on the assessment and the difficulty level of each of the questions. The resulting scores are then converted to performance levels, which are consistent with the Ontario Ministry of Education’s four achievement levels[2], generalized as follows:
- Level 1 represents achievement that falls much below the provincial standard
- Level 2 represents achievement that approaches the provincial standard
- Level 3 represents the provincial standard for achievement
- Level 4 represents achievement that surpasses the provincial standard
Following the assessment sessions, students respond to a survey that asks them questions about their language- and mathematics-related attitudes, perceptions and behaviours. Teachers and principals also respond to surveys that gather information about classroom and school demographics, the school learning environment, the use of EQAO data and instructional resources, teaching practices, educator collaboration, parental engagement, educator background and professional development.
In addition to individual student reports, the EQAO also generates school, school board and provincial reports that are meant to enable students, parents/caregivers and educators to support students’ learning, plan for improvements at both the school and system levels, and provide for accountability to the wider public. Primary and junior assessments are administered during a May-June time frame, and individual student results are available by the end of September. Sample individual student reports can be viewed within the framework documents (sources provided above). School, school board and provincial results can be accessed at: https://www.eqao.com/results/. The primary and junior assessments can be considered low-stakes, because the results do not count toward a student’s final grade. Educators, parents/caregivers and students can use the information to support children’s learning.
Grade 9 Assessment of Mathematics[3]
A total of approximately 140,000 students (English and French combined), enrolled in the Grade 9 de-streamed math course, are assessed annually on the knowledge and skills defined in the Grade 9 Mathematics Curriculum (2021). The Assessment’s blueprint can be accessed in the associated framework document (source referenced above). The assessment is made up of three components. The first is an introductory session in which students have an opportunity to experience a practice test that demonstrates the kinds of items that will appear on the actual assessment, as well as available online tools such as those described previously. The second component is the actual assessment, which comprises two sessions, each designed to be completed in one hour. Students are permitted to write the two sessions back-to-back in one sitting with a break in between, or they may write each sitting at different dates and times.
The assessment is made up, exclusively, of selected-response items such as drag-and-drop, drop-down menu, ordering, and single- and multi-selection item formats. The vast majority of Grade 9 items are operational; a relatively small number of field-test items are also presented to students. A Grade 9 sample test can be viewed here. The third assessment component involves student, teacher and principal surveys, as described above.
Like primary and junior math, the Grade 9 Assessment of Mathematics employs an msCAT model for test administration and IRT to calculate scaled/standardized scores, which are converted to the Ministry’s performance levels (described previously).
Two administrations of the Grade 9 assessment are available in the following assessment windows: January-February and May-June. Individual student results are available shortly after students have completed the test. A sample individual student report can be viewed within the framework document (source referenced above). School, school board and provincial results can be accessed at: https://www.eqao.com/results/. The Grade 9 assessment can be considered low- to medium-stakes, because schools and school boards can decide whether or not to count the results toward a student’s grade and by how much (up to 30%), in accordance with Ministry policy.[4]
Ontario Secondary School Literacy Test (OSSLT)[5]
A combined total of about 180,000 students (English and French) are assessed annually on the OSSLT, which reflects the expectations of The Ontario Curriculum for reading and writing skills across all subject areas to the end of Grade 9. The assessment’s blueprint can be accessed in the associated framework document (source referenced above). Like the Primary- and Junior-Division language assessments, the OSSLT uses a linear test design. As with the other EQAO assessments the OSSLT comprises both operational and field-test items. The test is made up of selected- and open-response items, which are administered in two sessions. The first session is designed to be completed in approximately 65 minutes; the second takes about 75 minutes. As was the case with the other EQAO assessments, the OSSLT may be taken in one sitting with a short break between sessions or on two different dates and times. An OSSLT practice/sample test can be accessed here. After the two assessment sessions have been completed, students complete a questionnaire that asks them about their literacy-related attitudes and perceptions.
IRT is used to determine a student’s outcome, which is either “successful” or “not yet successful.” There are two administration opportunities for the OSSLT: November and March-April assessment windows. Individual student results are reported after the test has been scored. A sample individual student report is shown within the framework document (source referenced above), and as previously mentioned, school, school board and provincial results can be accessed at: https://www.eqao.com/results/. The OSSLT is considered a high-stakes assessment, because successful completion of the Literacy Test is one of 32 requirements for Ontario secondary school graduation.
Benefits of the Transition
It is inevitable that transitioning from paper-based to e-assessment involves a great deal of planning, as well as some growing pains. This section of the article is not intended to be a comprehensive review, but rather it describes some important benefits that EQAO and the province have realized as a result of the transition to digital assessments.
Administrative Burden
The technology-based approach has eliminated many of the issues associated with paper-based testing. For years, educators at the school and school district levels had requested greater flexibility in how assessments were administered. In the past, the EQAO assessments were administered within two-week windows (except for the OSSLT, which was delivered on a single date). Printed assessment material was sent to schools; schools were responsible for keeping the tests securely stored; at testing times, the paper forms were distributed and collected from students; and then schools were responsible for packaging all test-related material for return to the Agency. This process resulted in a significant administrative burden for schools. EQAO online assessments have reduced that burden. Extending the administration windows to approximately one month means that schools have more flexibility to schedule tests and allows those that may lack large numbers of computers needed for the assessments to allocate them to groups of students over the longer time frame for test-taking purposes. Consequently, the Agency reports they have received no negative feedback from the field regarding a lack of technology to administer the assessments.
Fairness, Equity and Inclusion
From the outset, fairness, equity and inclusion have been hallmarks of EQAO assessments. The Agency has always included highly experienced educators from a range of social and ethnic backgrounds, gender, etc. in the design, development and scoring of the assessments, with the intent of ensuring that the needs of a diverse student population are addressed. With the advent of e-assessments, the Agency has established an even more thorough vetting process for test items from an equity, diversity and inclusion (EDI) perspective. EDI training is provided to all item writers, and a recently-created EDI Committee reviews all test items that have been proposed for use on the assessments. Furthermore, over the years, EQAO has strived to include as many students as possible in provincial assessments. To do this, accommodations and provisions have been provided for students with special education needs and English- and French-language learners whose needs were identified in their Individual Education Plans (IEP). Formerly, accommodations included the following formats in which the student receives the assessment (presentation format): sign language or oral interpreter; Braille; large-print booklets; coloured-paper (white, blue, green, yellow); and MP3 audio (including descriptive text) plus tactiles. Response formats for students included use of computer, word processor or assistive technology; speech synthesizer, Brailler, speech-to-text software, or alternative communication systems provided by the school board or school; and audio recording of student responses (accompanied by verbatim transcription of the audio recording.[6] The same accommodations are permitted for any students requiring paper versions of the assessments today, but in addition, the e-assessment system includes built-in text-to-speech and accessibility tools such as text reader, zoom in/out, line reader, high contrast, highlighter, eraser, calculator, as well as a “help” function for the test taker. In addition, following are some accommodations that are permitted for all students: quiet setting, use of headphones, supervised breaks, alternative setting and additional time.[7] The provision of such a broad array of accommodations ensures a high rate of student participation in the assessments.
To ensure the provincial assessments are fair, reliable and valid, EQAO has instituted: a Modernization in Measurement Advisory Panel, comprising an ethnically diverse group of academics (educational research and measurement), educators, school district personnel and social workers; an EQAO Advisory Committee made up of education stakeholders; and an EQAO Student Engagement Committee of students (Grades 7 – 12) representing all regions of the province. This latter committee is a source of student voice regarding their perspectives and experiences with the provincial assessments. All three of the Agency’s committees provide sources of important information for continuous improvement of the program.
Testing Environment
EQAO has always paid attention to creating a positive testing environment for students, so they can demonstrate knowledge and skills to the best of their abilities. Although some nervousness/stress is to be expected and can be beneficial in focusing students’ attention on the task at hand, severe test anxiety can be detrimental to the student and his/her ability to show what they know and can do. With the introduction of digital assessments, the Agency has taken a variety of steps to alleviate stress and test anxiety. Breathing exercises have been built into the e-assessment system to help students that may wish to engage in relaxation activities before taking the test. As mentioned previously, for each of the assessments (Primary and Junior, Grade 9 and OSSLT) an online sample/practice test is made available, which demonstrates the kinds of items that will appear on the actual assessment, as well as the online tools that are available. The provision of a help tool enables students to learn about the various assessment features and how they work. Accessibility features and accommodations (mentioned before), including the availability of extra time if needed, removes the stress that can result from a strictly timed test. By incorporating a variety of online item types, including gamification, student engagement in the assessment is enhanced. Another important feature of the assessments, which brings students a degree of comfort and confidence, is that they are permitted to use their own computers, provided they conform to the technical requirements of the assessment platform. The Agency reports that student questionnaire data suggests students are experiencing less anxiety when taking the assessments online.
Scoring and Reporting
Since all mathematics assessments are machine-scored, scorers are no longer required, and students’ results for the Grade 9 math assessment are made available the day after completion of the assessment. Although scorers continue to be needed for the constructed-response portions of the language assessments, the process is completed much more quickly and results in cost savings when compared to scoring associated with the paper-based approach.
Test Security
Compared to traditional paper-based tests, e-assessments improve security. For instance, the pressure of keeping test content secure before, during and after administration is less of an issue with online assessment. EQAO e-assessments, like many other online tests, come equipped with information technology methods and processes to protect the confidentiality, integrity and availability of computer systems; networks; and data against cyber-attacks and unauthorized access. Utilizing a lock-down browser disables navigation controls, so that students are unable to access other applications or websites to seek help in responding to questions. The linear and msCAT test designs, used by EQAO, reduce the likelihood of students sharing information about test content. All of these features ensure the integrity, validity and reliability of the assessments.
Conclusion
The work is never done! Since 2019, the Agency has collaborated with its assessment technology partner, Vretta, to modernize the assessment program. However, the state of technology in assessment is rapidly changing, and by its very nature, modernizing assessments means having to keep up-to-date with new developments in the digital large-scale assessment field. In the interest of continuous improvement, EQAO refers to the recently-published Guidelines for Technology-Based Assessment[8], an important reference for information about key factors and issues to consider when designing, administering and scoring assessments via digital platforms. In addition, the Agency continuously monitors emerging trends in the assessment industry and conducts assessment landscape scans of notable jurisdictions, worldwide, to ensure Ontario’s provincial student assessments continue to be state-of-the-art.
About the Author
Dr. Jones has extensive experience in the fields of large-scale educational assessment and program evaluation and has worked in the assessment and evaluation field for more than 35 years. Prior to founding RMJ Assessment, he held senior leadership positions with the Education Quality and Accountability Office (EQAO) in Ontario, as well as the Saskatchewan and British Columbia Ministries of Education. In these roles, he was responsible for initiatives related to student, program and curriculum evaluation; education quality indicators; school and school board improvement planning; school accreditation; and provincial, national and international testing.
Dr. Jones began his career as an educator at the elementary, secondary and post-secondary levels. Subsequently, he was a researcher and senior manager for an American-based multi-national corporation delivering consulting services in the Middle East.
Feel free to reach out to Rick at richard.jones@rmjassessment.com (or via LinkedIn) to inquire about best practices in large-scale assessment and/or program evaluation.
Acknowledgement
The author would like to acknowledge and thank members of the EQAO’s senior leadership team for their review of and helpful input to this article. If you wish to learn more about the EQAO/Ontario provincial student assessments, please refer to the Agency’s website at: https://www.eqao.com/, or direct inquiries to: info@eqao.com.
References
[1] Information in this section was accessed from the EQAO resources: Assessment of Reading Writing and Mathematics, Primary Division (Grades 1-3), Framework (2023) and Assessment of Reading, Writing and Mathematics, Junior Division (Grades 4-6), Framework (2023), located at: eqao.com/wp-content/uploads/2022/02/framework-grade3-assessment.pdf and eqao.com/wp-content/uploads/2022/02/framework-grade6-assessment.pdf.
[2] Descriptions of the Ontario Ministry of Education’s four achievement levels can be accessed at: https://www.dcp.edu.gov.on.ca/en/assessment-evaluation/levels-of-achievement
[3] Information in this section was accessed from the EQAO resource: Grade 9 Assessment of Mathematics Framework (2023), located at: eqao.com/wp-content/uploads/2020/10/Framework-grade9-assessment-of-mathematics.pdf
[4] Ontario Ministry of Education (2010). GROWING SUCCESS: ASSESSMENT, EVALUATION, AND REPORTING IN ONTARIO SCHOOLS, First Edition, Covering Grades 1 to 12.
[5] Information in this section was accessed from the EQAO resource: Ontario Secondary School Literacy Test Framework (2023), located at: eqao.com/wp-content/uploads/2021/01/framework-osslt.pdf
[6] EQAO. (2018). Administration and Accommodation Guide, Assessments of Reading, Writing and Mathematics: Primary Division (Grades 1-3) and Junior Division (Grades 4-6), 2018-2019, p.22.
[7] EQAO. (2023). User Guide for the Administration of the Ontario Secondary School Literacy Test, 2023-2024. This resource can be accessed at: https://eqao-kb-osslt.lswp.vretta.com/.
[8] International Test Commission and the Association of Test Publishers. (2022). Guidelines for Technology-Based Assessment. This resource can be accessed at: https://www.intestcom.org/upload/media-library/tba-guidelines-final-2-23-2023-v4-167785144642TgY.pdf.
_________________________________________
RMJ ASSESSMENT 