Perspectives on the challenges and opportunities for large-scale analyses on data use and data citations
April 3, 2025 | By: Clare DeanViews from Kai Li, Assistant Professor, School of Information Sciences, University of Tennessee
Please describe your previous research project(s) that related to data use, and the main findings from that research.
I was trained as a scientometrics researcher. One of my research interests is to understand how data and software are used in research, and how they influence scientific progress. To do that, it’s important to understand how to even find data and software in publications. Can we trace them through citations? Or maybe through full-text mentions? This is where my work on the topic of research data comes from.
During my PhD training, I looked at whether data are cited in publications (1); at the time, we found that this was still not a common practice – although there are differences across disciplines. Another line of my research has been on data papers. This is an interesting topic, because while they are research papers, they’re also describing data. The challenge to empirical research on this topic is that data papers are not identified as a separate document type or tagged consistently across bibliographic databases, as reported in our research (2). More recently, we also investigated if they are cited as research data, and found that most of the data papers are not cited as research data (3). So there is still an open question as to whether these data publications are indeed supporting data reuse, which is one of the reasons why they were designed.
What is your overall experience with data coverage in bibliographic databases, for the purposes of meta-research on data practices?
I used OpenAlex and Dimensions for my research recently. Many of the data journals are relatively new and may not have been sufficiently established in their disciplines; the newer databases, such as OpenAlex have better coverage of these publications.
A recent study looking at how datasets indexed in OpenAlex are cited prompted me to explore dataset objects in OpenAlex. I found that some objects tagged as datasets are not really datasets and there are many reasons behind the mislabelling. But this issue is not specific to OpenAlex, ‘what is data?’ is a big question to tackle for any quantitative study on research data. A solid operationalization of the definition of research data in scholarly records is definitely needed for us to understand what datasets are indexed in scholarly databases and if the indexing is complete.
What are your views on additional things we need to better identify and research data citations? What evidence on data citation practices do you think would be beneficial for the community?
For my own research, I am interested in how one dataset can be represented in many ways – a data paper, a record at a data repository, etc. Traditionally, a paper used to have just one citation format which did not change over time. But data (and software) can easily change, there are different versions and citable formats. This presents a challenge. I am interested in the connections between all these representations of the same dataset. It’s interesting and critical to trace the connections between those representations, bring them together, and understand how people cite them differently. There is already a lot of connection metadata created, but this type of data has not yet been fully incorporated into most bibliographic databases.
Make Data Count builds infrastructure and tools to capture data usage information, but we also want to bring change so that data is recognized in research evaluation. Do you view the visibility of data and data citations through indexing services as a step toward this? Are there other things that you think may help bring that change?
I would definitely say that having visibility could help with the general behavior of researchers across different communities. We still lack many pieces of infrastructure to transform data citation as part of the research norms in all research fields. It’d be interesting to see to what degree this contributes to what we want to achieve regarding data reuse and data citation. We may need to take baby steps, but we can progressively get there.
What is your motivation to explore the Data Citation Corpus?
We are really excited about the Data Citation Corpus dataset. The Corpus is broad and addresses a lot of the limitations of just using structured citation information to trace data citation, which we have known is limited. We strongly believe that this dataset can be used for many purposes in science of science or scientometric research. One of the examples that we recently discussed is the power structure behind the production and the use of data, i.e. who produced the data versus who uses the data. The Data Citation Corpus dataset can help to answer this.
What information and evidence do you think is needed to inform responsible data metrics?
That is a very big policy question. We need a top to bottom approach to developing policies that can lead to broader-scale changes in the community. There have been policies from funders, journals, and institutions about data sharing and data citation that are already playing a role. In my research, I’m interested in tracing evidence of how effective these changes are, and potentially inform future conversations about those policies.
And more broadly, we want policies to increase the accountability of scientific knowledge. Data sharing and data citations are part of the solution. Eventually, I believe one of the ultimate goals of the data research community is to support accountability, rigor and reproducibility in research.
References
- Zhao, M., Yan, E., & Li, K. (2018). Data set mentions and citations: A content analysis of full‐text publications. Journal of the Association for Information Science and Technology, 69(1), 32-46. https://doi.org/10.1002/asi.23919
- Jiao, C., Li, K. & Fang, Z. How are exclusively data journals indexed in major scholarly databases? An examination of four databases. Sci Data 10, 737 (2023). https://doi.org/10.1038/s41597-023-02625-x
- Li, K., Huang, P.-P., & Jeng, W. (2025). Are data papers cited as research data? Preliminary analysis on interdisciplinary data paper citations. Information Research an International Electronic Journal, 30(iConf), 1225–1233. https://doi.org/10.47989/ir30iConf46918
