‘Policies that make data count’: data evaluation in funder and policy-making processes

Blog post by Iratxe Puebla and Maria Alejandra Tejada Gómez. DOI: 10.60804/zb5q-qy55

The Make Data Count Summit last September brought together researchers, infrastructure providers and representatives across institutions, funding and policy areas to discuss data metrics and the responsible evaluation of data usage. Policy makers and funding agencies play a major role in driving forward open science practices and the adoption of open data as well as data evaluation, and so one of the sessions at the Summit focused on funder and policy-making processes. Mercè Crosas (Barcelona Supercomputing Center & CODATA) facilitated a discussion with Dagmar Meyer (European Research Council), Anna Diaz Font (Medical Research Foundation) and Maria Alejandra Tejada Gómez (Universidad Javeriana) about data evaluation and the role of data metrics to measure the impact of open data and inform decision-making in national contexts.

The need for data evaluation as part of open data policies

A number of research funders support or require data sharing. Dagmar Meyer and Anna Diaz Font both noted that the European Research Council and Medical Research Foundation stipulate as part of their policies that data from the funded research should be made available. Maria Alejandra Tejada Gómez shared that the policies approved under the latest National Development Plan in Colombia also expect access to the results and data from publicly-funded research. The panelists noted that an evaluation of data availability would fit within processes to check for policy compliance, but acknowledged that such compliance checks are currently rare.

As for any other evaluation process, there is a need to consider what is the purpose and scope of a data evaluation framework: is this at the researcher level or at program or national level? Is it to check for compliance or to inform funding or policy decisions? The metrics to incorporate into the evaluation will depend on the type of evaluation to undertake. When considering national policies, the framework will then influence policies at institutional level, where requirements for data management plans or data sharing are already implemented or in process. The institutional policies need to align to the broader national regulation for open data.

Current practices, challenges and opportunities

While the policies in place at different funders indicate that researchers may report any output as part of their grant applications or reports, in practice, few funders currently complete evaluations of data impact. This is also the case at institutional level, where policies have so far focused on the aspects of data up to the stage of sharing, with less attention beyond that point.

The discussion explored the challenges around the implementation of data evaluation in funder processes. The panelists raised a few elements that could do with further definition. Conceptually evaluation of data could be viewed in different ways: on the one hand, we have the aspect of the quality of the data and its accompanying metadata – e.g. use of persistent identifiers (PIDs), alignment to FAIR and CARE principles, compliance with privacy and security requirements – and whether the data shared is reproducible and reusable; on the other hand, there can be an evaluation of the reuse and impact of the data, that is, assessing whether others have reused the data and for what purposes. These two axes of evaluation will require a different combination of metrics, with the former possibly relying more on qualitative information and the latter more on quantitative metrics such as data citations, downloads, or the number of access requests. The metrics to implement may also differ depending on whether the evaluation is related to an individual researcher or rather a research program or a collection of outputs.

In addition to defining the metrics for each type of evaluation, there were also questions raised about the information required to provide adequate context so that panels would trust and incorporate it into their reviews. And as a complementary point, it would be important to have common ground on what the sources of data usage information would be, what tools are available to access the information, and what are the remaining gaps to address.

An aspect that garnered substantial comments was that of the resourcing needed to implement data evaluation in meaningful ways. The panel recognized that in order to complete evaluations of data reach and impact, there would need to be additional resources dedicated to curate and preserve the data to make that evaluation possible, to sustain the peer evaluation of data (with review panels incorporating the relevant skills to complete this type of assessment), and to gather and interpret quantitative measures collected through automated means or external sources. Ensuring a comprehensive view on the usage of data also relies on sustaining the infrastructure necessary to collect this information -e.g. repositories, data storage, interoperable systems and integrations-, and to make it openly available.

While this may seem challenging, there is a consensus that data is critical to address global challenges. We saw the critical role of open data in responding to the COVID-19 pandemic, and we should thus leverage the potential of data to accelerate research progress to respond to societal needs. To this end, there are a number of ways in which we can take steps to make data evaluation a reality. Many institutions have developed repositories and work with infrastructure providers that supply interoperable systems. It will be important to sustain this existing infrastructure while emphasizing interoperability to gain economies of scale. There are also tools and services that provide open information about data usage; while the usage measures may not be mature for their use in all areas of evaluation, we can already put the information available to use where this is meaningful. For example, the number of data citations is scaling and in combination with other information that funders collect, this can already provide insights into the reach and impact of datasets at program level.

Resourcing data evaluation processes is key

The discussion highlighted the importance that funding and policy-making agencies place on open data. The discussion touched on the need to identify the right information for a responsible and equitable evaluation of researchers and of research programs, and to resource data-evaluation efforts. Importantly, all panelists highlighted that we need a better understanding of how open data practices are evolving, and of the impact and return-on-investment of open data policies. And so, the time has come to match our recognition that data evaluation is important with the investment in resources necessary to make it happen.

We thank Anna Diaz Font (Medical Research Foundation), Mercè Crosas (Barcelona Supercomputing Center & CODATA), and Dagmar Meyer (European Research Council) for their contributions during the Make Data Count Summit and insightful discussions that fed into this post.