What Kind of Impact Are We Talking About?

In a February 2019 article, Andrea Nuzzolese and co-authors explored the viability of altmetrics as an assessment of research quality (“Do Altmetrics Work for Assessing Research Quality?” Scientometrics, 118(2):539–562; link.springer.com/article/10.1007/s11192-018-2988-z). They performed a study to determine any correlation between altmetric indicators and both citation count (as an article-level metric) and h-index (as an author-level metric). Interestingly, but perhaps not all that surprisingly, they found that the best correlation between altmetric activity and the citation-based metrics was number of readers in Mendeley. Mendeley readership also served to be the better predictor of future citations using a Naive Bayes algorithm.

While it makes sense that saving an article in a reference manager would in turn lead to increased citations in succeeding PRJAs, does this render other altmetric components irrelevant? For starters, at least one case study points to significant “dirty data” in Mendeley, which needs extensive cleanup to improve its accuracy as a component of the altmetrics tools (Bar-Ilan, Judit, et al. “Differences Between Altmetric Data Sources—A Case Study,” Journal of Altmetrics, v. 2, no. 1, 2019; doi.org/10.29024/joa.4).

Second, if we are measuring research output beyond PRJA and citation-based metrics, why should a correlation be tween alternative and citation metrics be necessary? Certainly, new formats of research are going to be impactful in ways other than cited references. Plus, in measuring media hits, tweets, blogs, and other popular public sources, alternative metrics stand to be a good start at demonstrating real-world or societal impact beyond the ivory tower. Remaining challenges include the difficulty in standardizing metrics due to the ever-changing availability of data sources and the need for additional research on the efficacy of altmetrics at capturing impact in not only scientific, research-oriented, and academic capacities, but also in this greater societal capacity.