Case Study
Remote AI Proctoring First Focused on the IT Certification Market
The product management team had cataloged the overserved and underserved features and functions of the current testing vendors. They created a roadmap for an end-to-end IT certification platform that streamlined the process of certifying vendor-specific IT professionals. The team and I put together branding messaging and marketing materials to communicate the value position. It was easy to differentiate Rosalyn. The systems that supported IT certification comprised a hodge-podge of general-purpose productivity apps and monolithic proprietary solutions. Rosalyn rolled them all up into an extensible platform with state-of-the-art automated proctoring. Communications relying on this core set of messages helped us secure pilots with the top 10 global technology companies and set the groundwork for expansion.
Differentiating Rosalyn from AI proctoring startups and traditional incumbents
Rosalyn is a software company providing artificial intelligence-powered remote exam proctoring to educational institutions and certification programs. Rosalyn’s remote AI proctoring is the core technology of their own end-to-end enterprise virtual assessment platform, integrating with learning management systems and test delivery engines. With Rosalyn, test sponsors can administer exams securely to anyone, anytime, anywhere in the world. Exams are invigilated in real-time by Rosalyn’s human-in-the-loop artificial intelligence technology. Hosted in the cloud and delivered by the web, Rosalyn’s cutting-edge technology offers an exceptional experience to test sponsors and test-taker alike.
The genesis of the Human/AI Proctoring Concept
The idea for Rosalyn’s assessment platform came out of years of experience conducting exams for demanding global clients such as the US Department of Defense. Delivering online exams remotely, the company encountered common non-independent test-taking (NITT) situations that threatened the validity and value of the assessments they were administering for clients. To support the assessment services, the company started development on invigilation technology that would maintain the integrity of the assessments while respecting the rights, privacy, and dignity of test-takers.
Rosalyn’s designed its solution to provide the same level of security as a test center but on-demand and delivered in the convenience and comfort of the student’s home or office. The company also strives to reduce the cost of administering exams so that high-quality, rigorous assessments are available to students worldwide. After surveying the market for invigilation of high-stakes exams, our research initially identified IT certification as the space with the most significant potential for growth, the need for services and resources to accomplish the mission, and the best market entry potential.
COVID-19 Changes the Landscape for Remote Proctoring
In February 2020, we could already feel the rumblings of what would be a seismic shift in the remote assessment field. I felt this shift in my own life when The College Board canceled my son’s SAT in March. No placement exams, course finals, or standardized testing were going on anywhere at the beginning of the pandemic. Although Rosalyn had tabled addressing the higher education and K-12 markets, the company was now getting calls from the most prominent test sponsors in the world. Taking stock of the environmental conditions, Rosalyn made a strong pivot towards a general-purpose invigilation solution for higher education and K-12.
Market’s Embrace of AI Proctoring
In this new market context of required social isolation, the benefits of AI proctoring ran deeper than cost-effectiveness or scale. They spoke to whether schools and companies could do any standardized testing at all. The incumbents in the space who had relied on physical test centers were unable to deliver solutions. Test sponsors who had dismissed remote testing and proctoring as having marginal value now looked to those same remote testing and proctoring solutions for relief.
In this new market context, Rosalyn had to:
- Differentiate from both the incumbent test companies like PSI and Pearson
- Differentiate from the upstart remote proctoring companies like Examity and ProctorU
- Give test sponsors a reason to believe in a startup for the mission-critical task of ensuring the integrity of their exams and ultimately degrees and honors conferred.
The Backlash Against AI Proctoring
Sometimes it pays to be late to the market. At the beginning of the COVID crisis, educational institutions picked up ad hoc remote proctoring solutions. Although the switch to remote learning was successful, remote testing at scale proved more troublesome. The remote proctoring solutions that schools put in place from established vendors engendered an immediate backlash from students, faculty, and parents around the issues of privacy, equity, and fairness. First, remote monitoring, in general, was viewed with suspicion by a large portion of stakeholders. The surveillance aspect of audio and video recording into the homes and bedrooms of students felt intrusive. Secondly, inequities in access to the Internet, computer equipment, and even quiet and private spaces in which to take exams exacerbated negative sentiment from students and the press. Lastly, questions about the accuracy and the definition of test violations and consequences concerned students, faculty, and stakeholders in the testing process.
Questions of Equity
Critics questioned the accuracy and fair treatment of incumbent proctoring companies’ AI monitoring technology. Research had shown that the facial identification algorithms created by the largest tech companies were distinctly biased against women of color. The media ran with stories, and most of the coverage of remote testing and proctoring was critical of the companies providing the services. This public sentiment presented an existential crisis to Rosalyn, whose technology revolved around AI computer vision.
Understanding Privacy Concerns
The first order of business was to get a handle on the problem. What did students think and believe about remote proctoring? What were their biggest complaints? Rather than hold focus groups or survey the population, Rosalyn established an industry-first Student Advisory Board (SAB). Although Rosalyn relied on a technical advisory board of leading lights in the EdTech and certification space, the student experience was opaque. Students’ attitudes and sentiments around privacy, diversity, equity, and inclusion increasingly drove market traction and acceptance of online proctoring solutions.
Customer Insights Point the Way Forward
The SAB gave us insights into what aspects of the remote proctoring were causing the most problems from the first meeting. A lot of the issues were not specific to AI but remote surveillance in general.
Transparency in AI proctoring
The most common complaint was a lack of transparency about how remote proctoring worked. Students wanted to know who had access to the audio and video recordings and what we did with the data during and after the test. Adjusting our communications and messaging about Rosalyn allowed us to assuage concerns of students and institutions based on misinformation. The most crucial fact about Rosalyn’s system is that there is always a human-in-the-middle of every proctoring decision. The AI may flag problematic behaviors, but the decision of whether that behavior is a rule violation is always in the hands of a real person.
Data Retention
Rosalyn also put in place concrete policies and business processes to minimize the collection and retention of data while obtaining consent from students for the use of data to improve and iterate on our AI models. We reinforced the idea that everyone involved in education has a mutual interest in fair and secure assessments. When we put remote invigilation in the context of the school’s range of actions to ensure that no one gets an unfair advantage, the relationship between Rosalyn and the students at our clients’ schools changed fundamentally.
Fairness in AI Proctoring
Many educators decry an epidemic of cheating in academia. The appearance of study sites like Chegg has allowed students to collaborate in ways never possible. The tools and resources on these sites facilitate students gaining an unfair advantage on tests if they choose. If students believe that their peers are violating the test rules to get a better grade, they are inclined to do the same.
It is crucial to the fair administration of tests that students know that the rules are enforced, violators will be detected, and consequences will ensue. Although ultimately, the rules and consequences of test security are the provinces of the educational institution, Rosalyn found opportunities to help further this goal, starting with being as transparent as possible with students before, during, and after the test about how the system monitors students.
Putting Students First
The marketing team at Rosalyn uses every user touchpoint to communicate the message about ensuring academic integrity with constant consideration of the rights of students and esteem for the learning process itself. The product team redesigned the user interface to clearly communicate how the AI worked and how students could follow the rules. In addition, we put together a massive content marketing strategy to promote the company’s vision. Through in-depth articles and guides, the company addresses the most significant concerns of students. Continual improvement of the accuracy and robustness of the AI models and diligence in protecting students’ privacy and data are also central to the effort. The paradigm shift to remote proctoring has already happened. By focusing marketing on the student experience, Rosalyn is well-positioned to become the leader in the category.