Introduction
Maintaining integrity is a key value of a rigorous assessment program. Incorporating quality assurance measures into test administration ensures that assessments conform to established guidelines and are valid, reliable, fair, accurate, and consistent. These measures guarantee equal treatment for all participants while maintaining the integrity and credibility of assessment results.
Ministries of education and testing organizations use a variety of methods to ensure that assessments are administered appropriately. This article describes the following combination of test-administration-related quality management processes undertaken by Ontario’s Education Quality and Accountability Office (EQAO):
- Standardized Administration Guidelines
- Training, Webinars, & Support
- Quality Assurance School Monitoring
- Attestation
- Assessment Security & Technology Controls
- Digital Monitoring
- Post-Administration Analysis (Data Forensics)
- Issues Reporting & Investigation
For context, readers should be aware that EQAO (the agency) develops and administers four types of provincial assessments in both English and French:
- Primary Division (Grade 3) assessments of reading, writing, and mathematics
- Junior Division (Grade 6) assessments of reading, writing, and mathematics
- Grade 9 Assessment of Mathematics
- Ontario Secondary School Literacy Test (OSSLT)
The agency also administers English and French versions of the EQAO Mathematics Proficiency Test (MPT), an evaluation required for teacher certification in Ontario.
Standardized Administration Guidelines
EQAO provides essential resources for each assessment to support standard administration in each assessment setting. Most importantly, it provides a detailed Administration User Guide containing information and instructions for educators, school and school board administrators, and IT professionals.
An example of a key administration guideline is that students must complete the assessment under standard conditions. For example, each session/stage must be completed in one continuous and supervised sitting. Schools may submit assessment sessions/stages once they are completed. Assessment sessions/stages are also closed automatically by the e-assessment system after one hour of inactivity and at 5:00 pm each day, providing additional assurance that students have standard access to the assessment.
The Professional Responsibilities section of the User Guide is of paramount importance. This section outlines the key professional responsibilities that school administrators, educators, and everyone involved in assessment administration must adhere to before, during, and after the assessments. The comprehensive list includes guidelines related to, for example:
- Reading and adhering to instructions
- Training for all those involved in test administration
- Technical requirements
- Scheduling, locations, and conditions of test administration
- Test administration protocols
- Breaches of security
- Student accommodations
The Administration User Guide also includes specific details about administration guidelines and the supports available for all students. It provides various resources, such as templates for letters to parents/guardians, checklists for teachers, and scripts to read to students. It also includes instructions for IT professionals; guidance on setting up and administering the assessments; information on what to do before, during, and after the assessments; and instructions for printing, scanning, and uploading responses for constructed-response formats.
Training, Webinars & Support
In addition to the User Guide, EQAO conducts webinars, delivers outreach presentations, and provides ongoing support to educators and administrators to help them understand the fundamental guidelines for appropriately implementing assessment policies, procedures, and accommodations, as well as navigating the modernized/digital assessments.
Throughout the school year, the agency’s School Support and Outreach Team works directly with teachers and administrators to enhance their understanding of the administration guidelines, assessment results, and their use.
Quality Assurance School Monitoring
Quality assurance (QA) monitoring of schools during test administration is another important component of EQAO’s approach to ensuring the quality and integrity of test administration. The agency engages an independent contractor (currently RMJ Assessment) to coordinate all related activities. EQAO selects a random sample of approximately 5% of schools, in both English and French, to be visited. In addition, some predetermined schools may be visited based on issues identified during previous assessment administrations. The agency strives to ensure that schools are not visited in successive years and are visited only once in a given school year, taking into consideration the provincial, national, and international assessments being administered.
The contractor recruits or sub-contracts Anglophone and Francophone retired school and school board administrators, as well as some former educators, to serve as quality assurance school monitors. The purposes of the school visits are to gather information about the extent to which assessments are administered in accordance with EQAO guidelines and to obtain feedback on EQAO assessment-related resources and procedures to identify areas for improvement.
The contractor is responsible for the following tasks:
- Developing, in collaboration with EQAO, observation and interview protocols in both official languages for use during school visits. The instruments include an observation and teacher interview form, a principal interview form, and an interview form for the staff member responsible for student accommodations and special considerations at the school.
- Contacting schools to obtain the dates within the assessment “window” on which they will administer the assessments.
- Training and orienting quality assurance school monitors.
- Scheduling school visits and supervising monitors.
- Collecting, compiling, and analyzing the data from school visits.
- Preparing reports on the extent to which assessments were administered in accordance with EQAO guidelines, feedback on assessment resources and processes, and recommendations to the agency.
Many jurisdictions do not conduct on-site audits during test administration due to resource considerations. Among those that do, some have a policy of arriving at schools on testing days without advance notice. EQAO’s policy is to notify schools and their respective school boards in advance when they have been selected for quality assurance school visits. The agency believes that effective communication and transparency in its processes help reinforce messaging about assessment administration guidelines and adherence to professional responsibilities.
Attestation
Before administering EQAO assessments, educators and administrators involved in the assessment process must adhere to and attest to EQAO’s professional responsibilities. Users of the EQAO e-assessment platform are required to review these responsibilities and attest that they understand and will follow them. The attestation includes:
- Reporting Security Breaches
- Not Deviating from EQAO Procedures and Guidelines
- Securing Testing Conditions and Assessment Content
- Ensuring Technical Readiness and System Controls
- Meeting Training Requirements
Failure to adhere to the attested guidelines may result in an investigation by EQAO and potentially the withholding of student or school results.
Assessment Security & Technology Controls
EQAO has implemented technology safeguards within its e-assessment systems. For example, Multi-Factor Authentication (MFA) is used for users with administrative access to strengthen account security and control access to assessment systems. Safe Exam Browser (SEB) or kiosk applications are used to prevent students from accessing external websites or applications that are not permitted during the assessment.
Teachers use specific SEB or kiosk passwords to unlock and close the secure browser on individual student devices. If a student with special education needs requires assistive technology, as identified via their Individual Education Plan, that is incompatible with the lockdown browser, schools are authorized to deploy approved Alternative/Direct Access accommodations. When Direct Access is used, alerts are provided to the supervising teacher if the student navigates away from the assessment.
A clear prohibition on additional devices in the testing environment, such as cellphones and smartwatches, is intended to prevent academic dishonesty and sharing of assessment content. A standard script is provided for teachers to outline these expectations. Secondary students must attest to the assessment security terms before accessing the test. In addition, clear instructions are provided stating that assessment materials must not be copied, saved, reproduced, distributed, published, or posted.
Digital Monitoring
Digital monitoring, in the form of live dashboards, is another way in which quality assurance in test administration is safeguarded. EQAO assessments provide school staff with live administration monitoring throughout the assessment. Through the EQAO assessment platform, school administrators and assigned teachers can view active, in-progress assessment sessions. They can monitor individual and group student completion progress, navigation alerts for students using Direct Access, session timing, as well as invigilator logs. Live monitoring can help identify technical issues or unsubmitted student tests while an assessment session remains open.
Post Administration Analysis (Data Forensics)
As part of its quality assurance framework, EQAO uses electronic assessment data, including time-stamped student activity and statistical analyses of response patterns, to detect anomalies that may indicate breaches of assessment security or administration protocols. Findings from these analyses may inform quality assurance reviews or investigations when the validity or reliability of assessment results may be affected.
Issues Reporting & Investigation
Reported or suspected irregularities in assessment administration are investigated by EQAO, in collaboration with its school board partners where appropriate. If an investigation determines that a breach has affected the reliability and/or validity of the results, student and/or school results may be withheld from reporting. Depending on the nature and severity of the matter, and where required or appropriate, it may also be reported to the Ontario College of Teachers.
Investigations may be triggered by:
- A reported or suspected security breach
- Deviations from EQAO administration procedures or guidelines
- Failure to maintain required testing conditions that allow students to work independently
- Other breaches of the Professional Responsibilities that could affect the validity or reliability of results
This process serves as an important quality assurance and integrity control within EQAO’s assessment program. As part of their professional responsibilities, school administrators and teachers are expected to report any issues to EQAO that may impact the security, reliability, or validity of an assessment.
During the assessment, issues are to be reported via the teacher invigilation page by using the “Report an Issue” button on the invigilation page of Vretta’s e-assessment platform. The reporting categories are Technical Support, Student Response Issue, or Administration Anomaly/Academic Dishonesty. Issues arising outside of the assessment platform should be reported to EQAO’s Information Centre.
Conclusion
Credibility matters! Assessments must be fair for all students. They must produce valid and reliable data that allow schools and policymakers to make informed educational decisions. Stakeholders, including students, parents, educators and the wider public, must also have trust and confidence that assessment outcomes are fair, accurate, and earned.
In keeping with best practices, EQAO implements systematic processes for establishing, monitoring, and enforcing standards to ensure the provincial assessments are valid, reliable, fair, secure, and appropriately administered. Collectively, these measures create a controlled testing environment that limits access to unauthorized resources, verifies device security, controls user access, and facilitates the investigation of potential breaches. These measures are designed to ensure that EQAO assessment results are fair, reliable, valid, and comparable across schools in Ontario.
The agency’s quality assurance measures in test administration are fundamental to maintaining the credibility of its assessment results. For more information about EQAO assessments, please visit the official website at https://eqao.com, or contact EQAO at info@eqao.com.
Acknowledgement
The author wishes to thank members of the EQAO’s Assessment & Reporting branch, as well as staff at Vretta, the technology partner of EQAO, for providing valuable information and guidance during the preparation of this article.
RMJ ASSESSMENT 