To support development of the Data Marketplace, we are working across government to implement a common approach to recording metadata.
What is the Data Marketplace?
Central Digital and Data Office (CDDO) in conjunction with Government Digital Service (GDS) are developing the Data Marketplace. Its goal is to provide a central place for government officials to find and understand how to access data held in other parts of government in a legal, ethical and effective way.
This is part of the DDaT Function’s mission of ‘Better Data to Power Decision Making’ as part of the Roadmap for Digital and Data 2022-2025.
Access to the right information on data sources is essential for empowering users across government to use data more effectively and transform the delivery of core services.
The mission to find the right data
For data to transform decision making, civil servants need to be able to find the data they need and assess whether it is fit for their purposes. Only then can they use that data to help make better decisions, and deliver better services for the public.
This is what we refer to as ‘data discoverability’.
The difficulties with data discoverability
Our user research has highlighted the difficulties encountered across government when trying to discover data.
Users told us:
- it takes considerable time to find potential data sources and their underlying data sets. In a couple of extreme cases it took users six months.
- civil servants then struggle to assess if these data would meet their needs because of a lack of appropriate information to describe these data, i.e. metadata.
Metadata is a set of data that describes and gives information about other data.
It needs to be adequate to allow users to assess if a dataset will meet their needs. This means including information about its provenance, what it contains, definitions and structure. It also needs to be represented in a way so that both humans and machines can process it.
There are published metadata standards to support data discovery, but a lack of standard methods of adoption means these standards are implemented inconsistently resulting in metadata that severely limits data discoverability. In order to work towards fulfilling the Data Marketplace vision, this needs to be addressed.
What we are doing
Preparing the ground for the Government Data Catalogue
Within the Data Marketplace, users will be able to ‘discover’ data via a Government Data Catalogue. This will link to departmental catalogues to harvest the required metadata to populate the central catalogue.
Departments that produce essential shared data assets across government, need to be recording core information in an aligned way for this to work.
CDDO were therefore tasked with developing, working collaboratively with departments, a common implementation approach to recording metadata; one that would be implemented within internal departmental data catalogues and then used, in turn, to populate the Data Marketplace Government Data Catalogue.
Developing a consistent metadata implementation approach
The cross-government approach we have developed is centred around 5 pillars reflecting the activities a metadata producer must work through:
- Specification
- Classification
- Accountability
- Implementation
- Publication
Our initial focus has been on Specification, which revolves around describing data.
We have developed a Metadata Requirements Specification, which builds on existing guidance on metadata standards. It includes:
- A list of ‘mandatory’ and ‘recommended’ attributes that should be provided for a data set within a data catalogue.
For example, all data sets should have ‘description’ which is fundamental to aiding discovery, and ‘date created’ and ‘date modified’ to enable a user to understand how recent the data are.
- Guidance on how to record attributes in an aligned way.
For example, data must be described with clear definitions and the use of terminology must rely on unambiguous language in its descriptions. This can be achieved by referencing terms and concepts from an agreed vocabulary or business glossary or those that come from applying a known standard, like the Data Catalogue Vocabulary (DCAT).
The DCAT is already used extensively across government, and our guidance recommends it is adopted for providing the core metadata attributes listed in the Metadata Requirements Specification. This is following extensive engagement, and in line with current use in government data portals in the UK and abroad.
Establishing a Metadata Implementation Working Group
Engagement and feedback has been central to our work in developing a common approach to producing metadata which is aligned to existing standards.
Our recommendations have now been endorsed across government, which is a vital step forward.
We are now moving forward with implementation, crucially at a time when many organisations in government and across public sector have implemented or started deploying their own data catalogues.
| For example, the Department for International Trade (DIT) created Data Workspace, which allows colleagues across the department to collaborate, share, and safely store their data. |
To ensure interoperability under the endorsed common approach, we have set up a Metadata Implementation Working Group (MIWG), with representatives from across government departments, representatives from the devolved administrations as well as external observers, like the Open Data Institute (ODI).
The group’s first task is to implement the Metadata Requirements Specification, aligned with the DCAT Standard, within their own data catalogues. This will form the basis of the Government Data Catalogue in the Data Marketplace.
Publishing guidance to support implementation
Parallel to this, MIWG members are providing valuable feedback and sharing implementation lessons; which will be used to populate a collection of guidance in the form of a Playbook geared towards Digital, Data and Technology (DDaT) professionals across government. It will include the Metadata Requirements Specification.
This will be a guide to developing metadata that meets the cross-government specification.
Longer term, MIWG will enrich the overarching cross-government Metadata Implementation Approach, providing further granularity beyond the core attributes listed in the Specification. This will likely include guidance on the provision of metadata attributes designed to assist users with evaluating a data set.
We expect published guidance to be also for the benefit of suppliers working with government who must already comply with the The Digital, Data and Technology Playbook. Implementing the aligned approach to metadata is fundamental to enabling data discoverability, and ultimately maximising the use of data across government.
How to get involved
We have made significant progress in gaining consensus across departments and the MIWG are driving forward with implementation. Feedback from members’ as they implement the specification will be incorporated to ensure the practicality of the guidance; MIWG will therefore continue to play a vital role. If you are responsible for metadata in your organisation you have a vested interest in shaping the approach and specification further.
We want to ensure views from all departments are represented in MIWG, so please get in touch if you would like to hear more or contribute to the group.
