Secure, Rapid, High Volume Data Integration with the SIF Infrastructure Specification
Executive Summary
The demand for data in education continues to skyrocket (Data Quality Campaign, 2022). In the real-world, however, constantly churning, time sensitive, high volume record sharing has historically been the Achilles’ heel of PK12 data integration efforts. When schools start to capture real time attendance events, the extra volume degrades and often completely overwhelms systems that rely on quick and timely data updates. Because of the extra load, updates to students and staff information could get lost in a sea of attendance events causing them to have to wait hours in line for processing.
To address these “digital ecosystem” challenges, the Access 4 Learning (A4L) Community released the SIF Infrastructure Specification (global) 3.4. This release is unique in the technical standards world in that it supports data management by both standardizing data and privacy on the “wire”. Previous versions of this blueprint are already in use in marketplace products and in schools across Australia, New Zealand, North America and the United Kingdom but the 3.4 Infrastructure version has a number of enhancements that allow real time information to more efficiently flow through “data pipes”.
The purpose of this white paper is to report on a third-party evaluation of these enhancements and present use-cases to demonstrate capabilities for handling high-volume data exchanges via “load testing”. The 3.4 Infrastructure Specification was tested to previous versions to compare performance. A data generator and a data subscriber were created to generate and push sample Student Daily and Period Attendance.
Using just the SIF Infrastructure Specification 3.4 created 20,000 times increase in throughput performance when compared to previous SIF 2.x Infrastructure Specification releases. The “hybrid” approach model, using existing SIF Infrastructure 2.x plus SIF Infrastructure 3.4 versions for those ecosystems who want to scale-up technologies, also showed improvements. By upgrading the receiving system, the result was 286 times increase in throughput performance than the traditional SIF integration. These results quantify “on the ground” implementation stories where states have updated their systems and discovered the increase in performance. These performance improvements enable greater data efficiency and opens the door to new use cases that require timely access to even more data.
The performance feature enhancements in the SIF Infrastructure Specification (global) 3.4 indicate a substantial increase in the “load” and “speed” in processing high volumes of PK12 data. In the future, it is expected that this standardized infrastructure will be used in conjunction with other data models, inside and outside of the education vertical, to allow for greater interoperability performance to support learning.