# Science of Science

All things science of science including scientometrics, measuring real world impact of scientific work and text mining scientific papers.  
  
Photo by Dom Fou on Unsplash: https://unsplash.com/s/photos/lecture?utm\_source=unsplash&utm\_medium=referral&utm\_content=creditCopyText

# Comprehensive Impact

Comprehensive Impact, as opposed to Academic Impact is an umbrella term for the impact of scientific work on society, economy and policy. Comprehensive impact covers a broad range of activities and is difficult - if not impossible - to define prescriptively. I coined the term Comprehensive Impact in my 2017 paper [Measuring scientific impact beyond academia: An assessment of existing impact metrics and proposed improvements](https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0173152)

### Why Should We Study Comprehensive Impact?

#### Why Scientists Care About Comprehensive Impact  


[Lots of papers write about the effect that the UK's focus on comprehensive impact affects the quality of research and individual researchers](https://hyp.is/K04A5G5YEe2Wv4uzp_WNKQ/viduketha.nsf.gov.lk:8585/FJDB_NSF/Science_and_Public_Policy/Vol.46(6)-2019/scz037.pdf)

#### Why Research Funders Care About Comprehensive Impact

[For research councils, being able to illustrate how their research impacts the economy and society helps them to compete for and justify their continued funding.](https://hyp.is/RL6rHm5TEe23HrM6ODp8xw/rf-downloads.s3.amazonaws.com/Kings+College+Report.pdf)

#### Why The Public Care About Comprehensive Impact  
  


#### Shortcomings of Academic Impact and Where Comprehensive Impact Helps  


Scientific papers are

### Related Definitions   


#### REF Definition of Impact

The Research Excellence Framework (REF) - a UK-wide scientometric instrument for research quality defines impact as:

> an effect on, change or benefit to the economy, society, culture, public policy or services, health, the environment or quality of life, beyond academia - [REF 2011](https://www.ref.ac.uk/2014/media/ref/content/pub/assessmentframeworkandguidanceonsubmissions/GOS%20including%20addendum.pdf)

The REF definition of impact is ambiguous as it does not specify the type of impact - I would argue that generating lots of citations is still impact but limited to academia. That is why we specify the term "comprehensive impact" when we talk about this concept.

# Tools and Metrics for Comprehensive Impact

### Research Excellence Framework (UK)

### ResearchFish (UK)

ResearchFish is a tool used by many funding bodies in the UK as a way to track outputs and impacts of scientific work. ResearchFish is used almost universally by all public funding bodies in the UK. ResearchFish primarily a data collection tool into which [the scientific investigators enter data about their work and corresponding outputs](https://hyp.is/aqWy9m5SEe2UlLdfYMDduw/rf-downloads.s3.amazonaws.com/Kings+College+Report.pdf) via a fine-grained question set. This information is fed back to the funding bodies. It is based on [early work within the RAND group led by Dr Steven Wooding.](https://hyp.is/go?url=https%3A%2F%2Fwww.rand.org%2Fcontent%2Fdam%2Frand%2Fpubs%2Fmonographs%2F2009%2FRAND_MG862.pdf&group=__world__)

In their [recent report ](http://rf-downloads.s3.amazonaws.com/Kings+College+Report.pdf)([mirror](https://wiki.jamesravey.me/attachments/41)), the authors describe their definition of Impact as being the same as that of REF's.

They use the word output to describe both academic and wider (or [comprehensive](https://wiki.jamesravey.me/books/science-of-science/page/comprehensive-impact "Comprehensive Impact")) impacts of research

[![image.png](https://wiki.jamesravey.me/uploads/images/gallery/2022-11/scaled-1680-/4uCimage.png)](https://wiki.jamesravey.me/uploads/images/gallery/2022-11/4uCimage.png)

Figure 1 From[ "Researchfish: A forward look." ](http://rf-downloads.s3.amazonaws.com/Kings+College+Report.pdf)Hinrichs, Saba, Erin Montague, and Jonathan Grant (2015).

### STAR Metrics (USA)