In CDDO, one of our aims is to support and enable responsible data sharing across government. The Data Marketplace is a key project we are developing to make this a reality. It will provide a front door to discover, access and share government data in a legal, ethical and trusted way, also offering supporting guidance and tools to help data sharing across government.
The cross government opportunity
Government ambitions and responsibilities often span a number of departments. For example, strategic policy objectives like levelling up and net zero, and responding to global crises, such as Covid-19, have a wide ranging impact, demanding cross-government collaboration to deliver appropriate solutions supported by the timely sharing of data.
Although this need for cross departmental data sharing is understood, we have found that many civil servants grapple with making sure they have the right data, at the right time, to make the right decisions - whether this is providing advice, researching a new policy, or delivering a new service. Our user research has uncovered it can be difficult to find out what data is held by other parts of government and it is unclear whether, and how, this can be accessed easily and in a legally compliant way.
To enable good data discovery and re-use, government departments also need to easily and confidently publish their data when it is available. Recognising that most departments are both suppliers and consumers of data, the Data Marketplace aims to create the right environment and conditions that bring data users and data suppliers together.
What will be available at the marketplace?
The Data Marketplace will not host or provide data directly. It will be a hub that adds value to existing government data services. This proposed solution, tested during our alpha phase, provides a centralised ‘front door’ on top of departmental data catalogues and internal data marketplaces. Its aim is to deliver the core, central services that are needed for great data sharing across government.
This philosophy has resulted in a design for the Marketplace based on four principal service areas:
- Discover: Providing the ability to find a dataset and assess its usefulness and allowing for information about shared datasets to be published for others to find.
- Share: Make agreeing and managing data shares easier, faster and compliant by using our guided workflows to design a data sharing agreement.
- Deliver: Support and guidance to create interoperable and reusable data services and Application Programming Interface (APIs) for data sharing.
- Guide: Knowledge and support for data standards, guidance and tools. Including insight from use of the Data Marketplace itself to provide an overarching view of the government data ecosystem.
Outlining the benefits
What does this all mean for government? Especially in relation to the ‘discover’ and ‘share’ components that will be flagship parts of our current beta programme.
The ‘discover’ service is underpinned by a catalogue that can be easily searched by users seeking data. Working with data suppliers, we are looking to populate it with metadata that provides sufficient information to allow users to find what they are looking for, and also allow them to dive deeper to fully explore whether the data can fully meet their requirements.
We do this all the time in our daily lives, for example browsing for clothes online and then drilling down to look at sizes, colours and delivery opinions to decide if we want to proceed. It's the same with data, and the better the information provided about the data, the less time is spent by users and suppliers sending emails backwards and forwards asking questions.
But what if you can’t find what you are looking for in the catalogue? What do you do next? We’re looking to cover this too, not in this current phase, but included on the roadmap is the provision of mechanisms in the Marketplace to surface data gaps and highlight new requirements.
Sharing data
Through the ‘discover’ services, when users find a ‘data match’, they will want to access the data as soon as possible. This is where the ‘share’ service comes in. Any use of public data by government needs to be legal and ethical, especially when the personal information of citizens is concerned. We must ensure we strike the right balance between requiring data to drive and improve services for citizens, while also using this data responsibly. Procedures exist to do this at a departmental level, however if we offer this current good practice as a single, consistent approach, we drive efficiency, transparency and set expectations of requirements.
The terms and conditions for data access still remain the responsibility of departments supplying the data, however the Data Marketplace will be able to help streamline data sharing request processes.
Working together to develop the service
To deliver this ambitious project, CDDO is collaborating with a number of government departments to collate different user and supplier perspectives, including Department for Education, Department for Environment, Food and Rural Affairs (DEFRA), HM Revenue and Customs, Office for National Statistics, and the Home Office. We are currently working with these departments to develop a private beta service that will go live in beta across government in spring 2024.
We are also continuing our user research with all departments and government organisations. We are keen to hear from anyone who would like to be included in our current or future research, particularly any data owners, data managers, data catalogue owners in the public sector or those building new government services. If you would like to get involved please sign up to take part in our user research.
Sharing data outside government
We know that the value of public sector data is wider than sharing across government. We are also working with the Department for Science, Innovation and Technology to understand how the approaches we are using for secure, legal and ethical sharing of public sector data across government, can be applied for similar sharing requirements across industry and other groups.
This is in response to Sir Patrick Vallance’s ‘Pro-innovation Regulation of Technologies Review - Digital Technologies’ report published in March 2023, which recommended government ‘facilitate greater industry access to public data, and prioritise wider data sharing and linkage across the public sector, to help deliver the government’s public services transformation programme.’
The project will look to engage with departments and industry to shape the future roadmap for the Data Marketplace and provide recommendations for any wider programmes of work in a final report published April 2024.
Ambitions for the future
The Data Marketplace will realise our ambition to make the process of finding and working with trusted data easier and more accessible. In turn this will help to make public services more joined-up, efficient and responsive to user needs. This is just the start, and the Data Marketplace's transformational benefits will start to be recognised when users engage with the hub. We will then be able to examine insights, to enable us to fully understand how the government data ecosystem functions - and how it can be improved.
DataConnect23
You can find out more about the Data Marketplace at three sessions during this year’s DataConnect, a conference dedicated to all things data. Please book a space via Eventbrite on:
- Monday 25 September, 10:00 - 10:45 - User-centred design and the Data Marketplace: making user-centred decisions.
- Wednesday, 27 September, 14:00 - 15:00 - The value of a cross-government Data Marketplace - This first session provides an update on the progress of developing Data Marketplace.
- Thursday 28 September 2023, 14:00 - 15:00 - The value of a cross-government Data Marketplace - This second session provides a further update.
To contact the CDDO Data Marketplace team directly, please email Teslim Abass
3 comments
Comment by Kwok Cheong Wong posted on
Will the market place cover transactional realtime/near-realtime event-driven data?
DWP has worked with GDS on a proof of concept for GDX, the platform that supposed to cover this area, and would be interested to be involved in any further work in this area.
Comment by Keiran Millard posted on
The simple answer is 'no'. The data marketplace is designed to compliment event driven data services such as GDX. The data marketplace satisfies the requirement for the pull-based data demand (largely based on the need to discover data) and GDX for a push-based data demand (where the need is established). In the marketplace you could however discover what push-based data services existed across government.
Comment by Kwok Cheong Wong posted on
Thanks for the information Keiran 🙂