A data platform is an integrated set of technologies that collectively meets an organization’s end-to-end data needs. It enables the acquisition, storage, preparation, delivery, and governance of your data, as well as a security layer for users and applications. A data platform is key to unlocking the value of your data. The data platform concept is primarily built on the basis of cloud technologies, as cloud-native data warehouses are now massively parallel-processed. Data pipelines can handle terabytes of data. Storage has become cheap and fast, and data processing frameworks can handle large volumes of data.
The ability to hold and process non-standard data outside of relational databases and add complexity and value to the data that is held, while Atifical Intelligence and Machine Learning offer powerful new tools to build on how the data is utilised. The challenge we face is that data silos that are often unscalable, contain duplicate, often out-of-date data, locked into proprietary solutions, and have no single security layer. A modern data platform tries to solve this problem. It’s a combination of interoperable, scalable, and replaceable technologies working together to deliver our overall enterprise level requirements for data.
The NDR data strategy has established a conceptual and logical architecture for the data platform required to support the objectives of the programme. In doing so it has identified the required capabilities of the architecture. A further engagement with Gartner assessed the available cloud providers using an evaluation framework based on these capabilities and our needs.
Based on this work and our internal investigations as well as our multi-cloud strategy we have procured access to Google Cloud products and services to fulfill key capabilities of the data platform. This project will deliver on the Data Platform and Data Management Architecture workstreams from the Data Strategy Roadmap. The project objective is to establish the NDR Data Platform that will support the capabilities described in the conceptual and logical data architecture in the data strategy and provide the environments and tools for organisations to acquire, organise, analyse, deliver, secure and govern health and care data.
This will be achieved through an architecture delivered through the Google Cloud Platform (GCP) and the project will consist of:
Workpackage 1 - Procurement of a reseller for Google cloud and consultancy services
Workpackage 2 - Design and commissioning of the GCP environment for NHS Wales
Workpackage 3 - Design and commissioning of data platform (including logical design and cyber security design and assurance processes, resilient network links)
Workpackage 4 - Service Management wrap for the Data Platform (including governance and Assurance for the platform and processes for on-boarding further technical solutions and data sets).
The project has seen significant progress and has completed a procurement exercise with a contract for a framework agreement in place with a reseller. Across the other workpackages there has been progress in identifying the services to be delivered by the project and how they feature in relation to the architectural view of the platform. Additionally, work is underway to establish the required teams to support and operate these services.
On the development side of the project work is underway on the piece of work to build and test the architecture using two main use cases, one focused on streaming data into the platform, converting into a FHIR profile and being able to process and analyse the data. The other use case is focused on a batch update of data which will also be processed, stored and analysed. Once the architecture has been suitably tested and the design confirmed the developments will be progressed into test and production environments in readiness for the completion of the assurance and governance work. These environments will be joined up with the NHS Wales network allowing for use by NHS staff as required.
The key benefit of the project is as an enabler, the NDR Programme seeks to provide many benefits in terms of data quality, data availability and even financial benefits from the work that it will deliver. All future benefits delivered by the NDR programme are made possible by having a single standards driven toolset to effectively manage and utilise the data that is held across the Health and Care spectrum in Wales.