Records Management as-a-service (RMaaS)
- Client
- Department of Finance
- Domain
- WoG shared services, ML
- Period
- Three months
- Role
- Prime Contractor
A whole-of-government records management platform that leveraged machine learning and microservices to automate classification at massive scale.
Challenge
Commonwealth agencies register around 230 million new records a year and sit on about a petabyte of digital records. The Department of Finance wanted to know whether one solution could serve all of them.
Research across 36 agencies found 80% of people dissatisfied with what they had. The reason was not features. Records management asked users to do work that had nothing to do with their actual job, so a great many records were simply never entered at all. Meanwhile disposal was still a manual process, which at this volume meant unwanted data piling up faster than anyone could clear it.
Solution
Rather than another records system, we proposed treating the problem as one of scale and classification, and using cloud machine learning and big data platforms to do the work the user had been doing.
The team built a records lifecycle microservice that creates records, manages versions, indexes and auto-classifies, all to National Archives standards and government information security requirements.
Then we tried to break it. The prototype ingested over 500 million sample records and held sub-second keyword search across the lot. Connectors for email, Windows file systems and cloud platforms including Twitter showed that any business system could feed the microservices rather than being replaced by them.
The wider architecture covered discovery and reuse of information assets through big-data search indexes and linked-data ontologies, unified search across multiple record sets, clustering and classification by machine learning, and merging records between organisational units. Together the microservices do everything the Records Management Act asks a system of record to do.
Outcomes
Half a billion records ingested, sub-second search across all of them, under sustained worst-case load that we set up deliberately.
That proved the thing that mattered: a single platform could take multiple petabytes of new records a year and stay available and secure. The design also moves the burden off the user, who gets to keep using normal systems while records management happens behind them. And because it is genuinely microservices rather than a monolith with an API, value-adding services can sit alongside the core functions instead of waiting for a vendor roadmap.
Technologies
Python with Django for the web application and Flask for the lightweight microservices. AWS S3, ECS, Elasticsearch and API Gateway. Machine learning for auto-classification. Linked-data ontologies for asset discovery. National Archives records management standards throughout.
See more of our workGoSource’s rapid prototyping helped us to prove the technical feasibility of our records management transformation program.
Robyn McLeod, Director, whole of government solutions Department of Finance