The releases of updated citation metrics by Clarivate and Scopus are happening around this time of year, so we would like to shed some light on these indexing databases and the appropriate usage of impact metrics. Constanze Schelhorn is MDPI’s Indexing Manager and we asked her to share her knowledge about some of the most important points.

Changes to citation metrics this time around

Constanze gives some thoughts on how the 2022 Impact Factors are different this year.

How does the 2022 Impact Factor release differ from previous years?

She notes, “The most apparent change this year: Impact Factors are displayed with just one decimal place, rather than three. Clarivate wants to encourage the use of additional metrics published in the Journal Citation Reports (JCR) and we believe that there is a trend in the research community to consult and compare with other metrics such as CiteScore, Source-Normalized Impact per Paper (SNIP), SJR.”

She goes on to say,

“Meanwhile this is the first year when Clarivate awards Impact Factors to journals outside of the two flagship indices, SCIE and SSCI, to include journals listed in the Emerging Sources Citation Index (ESCI) and the Arts and Humanities Citation Index (AHCI). This creates greater transparency for the full set of journals indexed in the Web of Science Core Collection.”

What does this mean for MDPI?

“For MDPI it means that 111 of our journals are awarded an Impact Factor for the first time in 2023, by the fact of being covered in the ESCI or AHCI. We appreciate the greater transparency created about the citedness of ESCI and AHCI-indexed journals, considering that many of these titles have a publication history of ten years or more.

While journals covered in ESCI and AHCI are now assigned Impact Factors, ranking and quartile information will only be provided for journals within SCIE and SSCI indexes.”

On ESCI indexed journals

“Journals covered in the ESCI index fulfill each of 24 quality criteria ensuring editorial rigor. A journal will be evaluated for inclusion in SCIE, SSCI or AHCI, respectively, using four impact criteria, once the citation activity indicates its citation impact puts it in an influential position compared with peers, i.e. above average in its field.

Impact evaluations for ESCI journals have been on hold since 2022 for all publishers across the board. Likewise, journals applying for inclusion in Web of Science, if selected for coverage, will only be added to ESCI. We expect evaluation activity for emerging titles to pick up again in 2024.”

Why are citation metrics released at this time of year?

She explains, “Indexing and curating publications is time-consuming. It takes at least until February or March to ensure all articles published in the previous calendar year have been processed properly within databases. That leaves two more months for further data verification by the database specialists before the citation metrics are then finalized–by the start of June for Scopus and towards the end of June for Web of Science.”

Caveats around the use of Impact Factors

There are always some factors to consider when discussing and reviewing details about Impact Factors.

Impact Factors remain incredibly popular. Are there any risks if their limits are not well understood?

“As a publisher we are well aware that scholars care quite a bit about a journal’s Impact Factor. Surveys show that IFs surpass most other factors in importance, including the APC amount, or whether an article is published under a Creative Commons license.

There are certain risks associated with misuses of Impact Factors. Firstly, unlike the equally common h-index, which is used as a citation metric for individual researchers provided by different sources such as Google Scholar, Scopus or Clarivate for example, Impact Factors are journal metrics and should be used as such. They say little to nothing about the influence of a single article in a given journal.”

About citation frequencies

Constanze explains, “[C]itation frequencies of articles in a journal follow an asymmetrical, strongly right-skewed distribution, starting from zero cites as you’d expect and with no limit to the upper end. Taking the average of such a distribution, which Impact Factors boil down to, can give disproportionate weight to outliers, i.e. a few very highly cited articles, oftentimes review articles, which tend to be cited more frequently compared with original research papers.

To circumvent the issue of Impact Factor being driven by outliers, some years ago Clarivate added median citation counts per review or research article in a journal to the journal data, giving a more realistic benchmark for article impact. Yet Impact Factors still grab all the attention, making it a necessity to educate people about their appropriate use.”

More caveats for Impact Factors

“[A]nother caveat around Impact Factors: they are not normalized by research fields, meaning that articles in fields with high rates of citations (think clinical medicine or biochemistry) tend to be cited more frequently compared with fields where citations are more sparse (think mathematics or anthropology).

That’s obviously no issue when comparing journals within one specific field, but turns out to be problematic for organizations aiming to evaluate and benchmark the scholarly impact of journals across disciplines. Within the JCR, the more recently introduced Journal Citation Indicator is a field-normalized metric that allows comparing impact across fields.”

Rankings

To address some of the questions regarding Impact Factors, Constanze helps us to understand some of the differences.

What does it mean for a journal to be ranked in “Q1” or “Q2”?

“Journals listed to be in Q1 of a ranking are in the top 25% of the distribution of citation metrics within a given subject category. In other words, its rank is in the 75th percentile or above, higher than at least 75% of journals in that category. Similarly, journals ranked Q2 have an IF which is in the 50th to 75th percentile of the category distribution.

You might have noticed that it’s the opposite to common usage in statistics whereby the 25% lowest values generally make up the first quartile (Q1) and the 25% highest make up the fourth quartile (Q4).”

Impact Factor decimals

“As a consequence of rounding IFs to one decimal place, ties are more likely to occur. Clarivate thus states that “quartile distributions are calculated based on the number of rank positions within a specific category, rather than simply dividing the number of journals evenly into four. Therefore, it is worth noting that quartiles are not of equal size when considering the number of journals in each category.”

Now almost one third of the more than 20,000 journals indexed in the Web of Science are assigned to more than one category. They are given a rank quartile for each category to which they belong.

Looking at 2022 CiteScores in the Scopus database, 80% of MDPI journals have a score that ranks them in Q1 or Q2 in at least one subject category.”

Alternative metrics

Impact Factors aren’t the only metrics that can be used. What do they mean and how can they be used to understand more about journals?

What about some of the alternative influence metrics that are available in scholarly publishing?

“Citations provide a signal of academic impact, a nod from one researcher (group) to another. More immediate signs of recognition, novelty, and (dis)approval exist in the digital space. Sharing an article on social media platforms, or mentions of the latest research by news outlets, or uploads to video platforms.

The platform Altmetric monitors a host of alternative sources to provide attention and engagement scores that reach well beyond citations. The colorful doughnuts capturing the scores are shown in the upper right corner on article pages–a useful alternative to article download and citation stats.”