Digitising NAPLAN: The transformation of Australia’s national assessment program.

Digitisation of a nationwide public benefit project remains a lofty goal today, and in 2015, when the partnership between ACARA, ESA and Janison began, it was even more so.

There would be zero tolerance for failure with such a large and important public undertaking. So Janison needed a cloud partner with the same professionalism and quality. That partner would be Microsoft, bringing its ability to host a proposed cloud solution in regional Azure data centres.

This idea originally took shape to the credit of the visionaries in the room back in 2015. It became the incredible journey we know today as the digital transformation of NAPLAN.

Students of the early 2000s and prior may recognise this typical conversation on leaving an exam room:

“Did you find that hard? I thought I wasn’t going to finish on time!”
“No, I found that one easy. I had nothing to do for the last 30 minutes.”

Perhaps these two theoretical students achieved the same results. Yet, no educator would know that one may have needed a different level of support from the other.

Years before digitisation, paper-based school examinations clearly had inherent limitations when it came to providing a more nuanced grasp of what students knew or didn’t know – for both high and lower performers. With paper tests, every student sits the exact same assessment. Therefore, it was necessary to pitch NAPLAN’s test questions at the rough centre of students’ capability, with some extension at either end.

Add to the fact that students were already using many more digital devices in their everyday lives: texts or instant messaging over writing or calling, streaming content over watching television, etc., to the point that educators could no longer ignore their modern preferences.

ESA Chief Executive Andrew Smith explains: “It’s clear that young people are more engaged by technology than by pen and paper. Therefore, we understood that we could better engage students in the test using technology.”

Modernising student engagement was just one of the objectives for the digitisation of NAPLAN, which was first introduced in 2008 to consistently monitor literacy and numeracy levels of Year 3, 5, 7 and 9 students. As years passed, the limitations of manual exam delivery grew, falling into a number of broad categories:

• Equity: Equity of access is imperative for a national testing program, especially in a country as colossal and diverse as Australia. That said, its varying degrees of internet bandwidth meant equity would not necessarily be achieved with digitisation either. It would need bespoke solutions instead.
• Performance: The turnaround times for NAPLAN results had always been frustrating for schools, parents and students. It could take up to four months for students to receive their results from the pen-and-paper format of the test – delaying teachers’ ability to plan and implement targeted teaching interventions.
• Security: Paper tests are vulnerable to breaches in security protocols, such as the papers falling into the wrong hands or test booklets going missing in transit.
• Scalability and futureproofing: To futureproof the testing system, it needed to scale to meet increasing student populations emerging test requirements, not to mention avoiding the need for a costly reconstruction further down the line.

All these factors became critical drivers for digitisation.

Of course, such a big change for the nationwide schooling system called for a substantial effort that required cooperation across many parties. Eight jurisdictions, states and territories, plus the Commonwealth, would need to agree on key aspects of the program collectively – or else it couldn’t move forward. And while this undertaking would be huge at the system level, schools’ individual change management processes would arguably be more significant.

Still, a countrywide NAPLAN examination system that was engaging for students and able to adapt to the styles and aptitudes of each was highly compelling. So ESA, with the support of Janison and Microsoft, forged ahead.

ESA put out feelers for an organisation capable of delivering a digital NAPLAN test, and ended up selecting Janison as its partner. Together they engaged multiple jurisdictions to gather functional specifications and agree on priority processes. Educators and governments had already trusted Janison’s digital assessment platform to deliver, mark and analyse tests for millions of test takers over the years, so using the platform as a customisable base seemed perfect.

Providing students with a consistent, usable test experience was top priority. But Australia’s states and territories already had their own independent, varied NAPLAN delivery processes, which made this challenging. The goal was to unite and unify these processes so that students could have consistent, smooth test experience without interruptions or distractions.

Equally, the project’s success hinged on individual schools upgrading their internet connectivity, local infrastructure and hardware to handle many simultaneous students. They would also need to train administrators and implement programs to raise digital literacy among school staff. So a big part of the challenge would be coordinating test-ready preparation nationwide.

“So as much as there was great excitement about the opportunity, there was certainly an understanding that this was not going to be a simple task because of the amount of change it might require across education at all levels,” says Smith.

Rising to the challenge, Janison sat down with its technology partner Microsoft to architect a platform that would meet NAPLAN’s current needs and scale for the future – whatever that might bring. “Most solutions get re-architected and retooled every couple of years. For us, it’s a constant process with Microsoft to figure out what will be the next big technology development and how we should incorporate it,” says Welsh.

Scalability is a big part of that. With populations increasing and digital tests evolving, a digital NAPLAN must be able to scale seamlessly without refactoring the underlying infrastructure. “We used around 35 different types of Azure services to build the NAPLAN Online platform. One of those is a microservices layer that allows us to achieve elastic scale to support NAPLAN far into the future,” Welsh says.

“We leaned on Microsoft’s capacity for hyper-scale clouds that could enable the heavy lifting required for NAPLAN. And we tested far beyond current expectations for concurrent student access, to 450,000. In some cases, we went even further, testing to 600,000 while leveraging the scalability of Microsoft Azure to deliver that sort of volume. We can’t think of anybody else in the world managing that volume of simultaneous delivery for a school exam,” he adds.

Aside from delivery, there was test content to consider.

The Australian Curriculum, Assessment and Reporting Authority (ACARA), is responsible for developing, delivering and reporting on the national curriculum and national assessment program (NAPLAN) for Australian students.

Digitising NAPLAN provided the opportunity to do more than simply lifting and dropping current tests into an online platform. ACARA had the chance to modernise the test format significantly, with Janison needing to understand key factors like test design and security requirements – are a top priority in education delivery.

ACARA was most excited by the prospect of an adaptive test, capable of responding to the answers students gave by serving them a set of questions that were either a little harder or easier, depending on how they were progressing. NAPLAN could achieve adaptive testing via an algorithm embedded within the platform software that intelligently adjusts the difficulty of test questions by interpreting answers at crucial points.

“There are always students breezing through their test, completing it in no time and getting everything right. Yet, we don’t know what they’re fully capable of. Adaptive testing allows us to stretch those students by serving them more challenging questions to get a better understanding of exactly what their level of attainment can be,” says Smith.

“Just as importantly, there are plenty of students who, for various reasons, might not answer a single question on a pen-and-paper test, or they might answer just one or two. The adaptive test engages them far more than pen and paper and boosts their confidence by directing them into an easier set of questions if that’s what their responses indicate.”

Moving NAPLAN online presented a huge challenge: what happens if a school’s internet connection drops during the important tests?

Anyone who has neglected to switch on autosaving of a Microsoft Word document can relate to the stress of losing unsaved work. However, loss of work is unimaginable for an important national assessment event like NAPLAN. Therefore Janison, on behalf of ESA, made test recoverability their number one priority when architecting the NAPLAN Online solution. This meant adding protocols for auto-saving students’ test responses offline every roughly 30 seconds, so that in the event of an internet dropout, only seconds of work would be lost (if any at all).

Meanwhile, other schools in more remote parts of Australia suffer from a near-permanent lack of internet connectivity. So being able to develop the low bandwidth solution for these school was crucial.

In such a situation, Janison’s ‘low and no bandwidth’ test player continues to run students’ exams even if the school’s internet or network connection drops. When the connection restores, the tests data then uploads and syncs with the system, having been securely stored in their browser. This crucial feature gives students an uninterrupted, distraction-free experience no matter their connectivity – no freezing. As such it not only protects schools against data loss but fosters equity, providing a level playing field for test-takers in rural and regional areas which have less dependable internet coverage.

“Somebody might dig up the internet pipe that supplies a whole lot of schools by accident, or there could be outages from various providers at different times, causing computers to drop off their local Wi-Fi network and reconnect later in time. So because there was an understanding that this could be the case, the system was designed to upload information every 30 seconds. Should something go wrong, you lose no more than 30 seconds of students’ work,” says Smith. “Students can pick up where they left off.”

This feature would become a significant driver of equitable education in lower or patchy connectivity areas, enabling them to benefit from a digital test solution.

Two years of planning and building resulted in a staged rollout, which acknowledged that some schools were more ready than others to embrace NAPLAN Online.

Smith says: “The thinking behind a staged approach was that we could build confidence among stakeholders in the test itself, but also different schools or different stages of readiness. Some schools were equipped to come online early, so we wanted to give them that opportunity. Others required more time to build their infrastructure and confidence to the point where they could undertake the online test seamlessly.”

Here’s how the staged rollout looked, stage by stage:

2016: In 2016, Janison entered into a seven-year agreement with Education Services Australia (ESA) on behalf of the Australian Federal Government to become the software platform provider for the online version of the National Assessment Platform – Literacy and Numeracy (NAPLAN), a national, annual assessment for all school children in Years 3, 5, 7 and 9.

2018: In a world-first, in May 2018, the platform successfully delivered the NAPLAN Online assessment to 1,400 schools with a total of 200,000 school students over a nine-day assessment window nationwide. Speaking at the time of the test delivery period in May 2018, Janison’s then-head of operations Sandy MacDonald said: “An assessment event such as this requires a highly scalable solution, developed in a way where no individual computer is overloaded. We’ve spread the load-balancing across disparate nodes of computers.”

Welsh reflects on this crucial moment in the project: “Our first measure of success at that point of going live for the first time was whether students could log on and navigate tests consistently, without the system slowing down. That success happened.

“Many had thought what we were aiming to achieve impossible, but we proved them wrong with NAPLAN Online, showing that a platform of our scale can work in the assessment space. I think we all breathed a sigh of relief and gained the confidence to pursue our plans for a countrywide rollout.”

2019: In 2019, Janison continued the momentum, delivering NAPLAN Online to approximately 660,000 students, with the remainder of students continuing to take the test on paper. This hybrid approach and measured rollout enabled the consortium to monitor the transition for authenticity.

The subsequent decision by education ministers to cancel NAPLAN in 2020 due to COVID-19 allowed more time for Janison, in partnership with ESA, to work on improving and strengthening the platform in readiness for NAPLAN 2021.

2021: In May 2021, still in the thick of the pandemic, approximately 70 per cent of Year 3, 5, 7 and 9 students sat the new digital NAPLAN tests. That’s 800,000 school students and more than three million tests, all completed within two weeks. Janison supported more than 195,000 concurrent NAPLAN tests at its peak, with more than 32,000 transactions per second.

2022 and beyond: Test statistics in 2022 set a record for peak concurrent student numbers – believed to be larger than any other online school assessment globally. In the second week of May 2022, the platform delivered 4.3 million tests (1m+ in a single day) for 1.2 million school students. At its peak, more than 315,000 students simultaneously sat the exam without issue. This has reaffirmed Janison’s market-leading position of seamlessly delivering the highest level of security, reliability and scalable infrastructure for online exams in an equitable and accessible manner for all students.

In fact, ministers had planned the 100 per cent milestone for 2021, but COVID-19 and lockdown conditions pushed the date forward by one year. It was an acid test of both the digital platform developed by Janison and the Microsoft Azure cloud computing platform. Educators everywhere held their breath before learning the solution had passed with flying colours nationwide.

While it’s early days in the 100% nationwide digital life of NAPLAN, the collaboration has brought welcome results.

Algorithms built into the NAPLAN flow make each test adaptive, allowing every student to give their strongest effort on the day because the assessment is serving up appropriately challenging questions based on their answers. “Janison was founded on learning management systems, and we built some of that quizzing technology within the platform,” says Welsh.

Accessibility is another critical equaliser. NAPLAN Online was designed for multiple accessibility challenges, even accounting for those students who are more comfortable using an iPad than a laptop. Artificial intelligence will become key in helping ACARA understand the impact of those accessibility measures and refine them for the future.

“It’s not just about providing for people who might have visual impairments, different colour perception and other access needs such as Braille keyboards. It’s also about having the same privileges as others. For example, an Indigenous school in remote Australia may have low device numbers and poor connectivity. We designed NAPLAN Online with those challenges in mind,” says Welsh.

For example, Janison’s low and no bandwidth solution is proving essential to the ability of rural and regional schools to participate in NAPLAN in times of unreliable connectivity. It has made all the difference to some schools whose remote location had even prevented them from participating in the paper version of NAPLAN.

“This all comes back to equity, meaning young people all having the opportunity to do the same test under the same conditions so that we get a better understanding of what they can and can’t do and give teachers that guidance about what the next learning steps for those young people are,” says Smith.

The risk of physical exam booklets going missing in transit has also been resolved. Lost mail was always a possibility wherever there were manual transport logistics, and the risk is higher for schools with long and multi-phase mail journeys. Students are now not only assured of an uninterrupted test experience and safe treatment of their academic records, but also a shorter wait for their results being returned to schools.

Seamless performance is vital in test scenarios, especially one so important as NAPLAN. The ultimate performance measure is that the platform continues to operate well throughout the exam period. With NAPLAN now digitised, students can sit tests online on a range of devices, wherever they are based. Microsoft Azure locations around Australia help avoid latency issues, and there are measures in place to ensure test continuity and prevent data loss.

Speaking of the procedures for managing any incidents, including internet outages, ACARA says:

Another key performance indicator is more granular data and results, delivered at speed so that schools are not waiting for critical teaching direction. Automated scoring of all but the writing test means ACARA can produce results and report back to teachers in a few weeks versus a few months. “From next year, the expectation is results will be back with the schools within a month,” says Smith.

Speed guarantees have enabled ministers to move NAPLAN tests forward from May to March, commencing in 2023. “Bringing the test forward into Term One allows even more time for teachers to work with the results,” says Smith. “We probably wouldn’t have done that before due to logistical challenges. For example, distributing test papers to remote parts of the country during the wet season.”

Due to the incredibly sensitive nature of data and privacy concerns relating to young students, the assessment platform needed to meet rigorous privacy and security requirements. The NAPLAN Online platform complies with Europe’s General Data Protection Regulation (GDPR), the Australian Government’s Information Security Manual, and the ISO 27001 information security standard.

“Delivering NAPLAN involves the collection and distribution of student test results on a significant scale. I think protecting young people’s data is essential, and much work goes into securing that information,” says Welsh, “so that means we’re entrusted with a monumental responsibility to safeguard this.”

Equally important was ensuring test submissions could not be lost or tampered with. Janison Insights’ in-built audit trail for recording every touch point would effectively eliminate the human error element of misplaced papers.

As with any major change in process there are additional impacts that sit outside of primary priorities. In this case it was environmental benefit. A high-level audit of paper saved and environmental cost avoided through lack of transportation presented impressive numbers – explained here by Welsh.

The incredible undertaking of transitioning NAPLAN online yielded several learnings. “I think we’ve learned the value of being more flexible in designing and delivering a complex program like this,” says Smith. “It also reminded us of the critical importance of people working well together in the same space. And I think we learned how to work remotely while still collaborating closely.”

“NAPLAN Online is not a technology program. It’s fundamentally an education program,” says Welsh. “It’s about understanding what young people in Australia can and can’t do regarding their education. It assesses their knowledge and skills, allowing teachers to work directly with that understanding. Bringing NAPLAN online opens the door to all sorts of possibilities regarding how we give young people and their teachers even better information about students’ literacy and numeracy.”

The most meaningful marker of NAPLAN’s digital success is the feedback received from state and territory education departments, whose schools report an overwhelmingly positive experience for students and teachers alike.

Wendy Powson, principal of Lilydale High School, reported that her school had an encouragingly positive experience with the online tests:

“It took an awful lot of organisation, as you can imagine. We had to make sure kids who didn’t have devices had one,” she said. “What we are looking forward to is having the results coming back a bit early, so it can inform teaching.”

Principal of Mount Druitt School, Glenn Patchell, said the school, located in Sydney’s outer western suburbs, never let its postcode determine its success.

“We still have high expectations of the kids, we have focused teaching, we have engaged learning,” and “we always use the data to inform our teaching.”