The need for a new set of measures to assess the impact of research in earth sciences in Indonesia

Background: Earth sciences is one of those sensitive field sciences that are closely needed to solve local problems within local physical and social settings. Earth researchers find state-of-the-art of topics in earth sciences by using scientific databases, conduct research on the topics, and write about them. However, the accessibility, readability, and usability of those articles for local communities are major problems in measuring the impact of research, although it may be covered by well-known international scientific databases.
 Objectives: To ascertain empirically whether there are differences in document distribution, in the proportions of openly accessible documents, and in the geographical coverage of earth sciences topics as revealed through analyses of documents retrieved from scientific databases and to propose new measures for assessing the impact of research in earth sciences based on those differences.
 Methods: Relevant documents were retrieved using ‘earth sciences’ as a search term in English and other languages from ten databases of scientific publications. The results of these searches were analysed using frequency analysis and a quantitative- descriptive design.
 Results: (1) The number of articles in English from international databases exceeded the number of articles in native languages from national-level databases. (2) The number of open-access (OA) articles in the national databases was higher than that in other databases. (3) The geographical coverage of earth science papers was uneven between countries when the number of documents retrieved from closed-access commercial databases was compared to that from the other databases. (4) The regulations in Indonesia related to promotion of lecturers assign greater weighting to publications indexed in Scopus and the Web of Science (WoS) and publications in journals with impact factors are assigned a higher weighting.
 Conclusions: The dominance of scientific articles in English as well as the paucity of OA publications indexed in international databases (compared to those in national or regional databases) may have been due to the greater weighting assigned to such publications. Consequently, the relevance of research reported in those publications to local communities has been questioned. This article suggests some open-science practices to transform the current regulations related to promotion into a more responsible measurement of research performance and impact.

Unfortunately, by actively participating in the game of world rankings and trying to be considered a world-class university, universities strongly encourage their staff to publish in English, because the criteria considered for assessment and rankings are typically drafted by, and beneficial to, the so-called WEIRD nations (short for Western, educated, industrialized, rich, and democratic). 2 In Indonesia, this bias might lead to inadequate application of research results to satisfy the public's needs and, ultimately, to waste of the money and resources spent by the state on such research. 3 The responsibility of research to contribute to community was also recently emphasized by the Indonesian president and the Indonesian Minister for Education and Culture and now features in the draft policy -to be implemented in the near future -related to the careers of lecturers and performance of higher education. 4 The primary research question addressed in the present study is this: Do we need a new perspective in measuring the impact of research? The conventional measures of such impact apply at three levels, namely at the level of the journal (the impact factor or the SCImago Journal Rank, for example), at the level of the article (the number of times the article is cited, for example), and at the level of the author (h-index, for example). These metrics form the basis of criteria widely used to measure the reputation of an institution as well as that of authors and research groups.
Indonesians tend to read and cite many Indonesian articles; therefore, papers written in English gather fewer citations. 5 Secondly, although cumulative citations of articles published in a journal increase its impact factor, the number of citations won by individual articles may not correspond to the impact factor of the journal that published those articles; therefore, it is inappropriate to measure the quality of an article based on the impact factor of the journal.
Many Indonesian researchers are unaware of the open citation movement being championed at the global level 6 and may even wilfully ignore the fact that commercial databases, promoted by aspirations to be a world-class university, implement a closed citation system with unfavourable effects on publicly available knowledge and services. 7 How can we expect to benefit from citation-based evaluation of research if the characteristics of citations and citation analysis -the main criterion for evaluationare not well reflected by the academic community, let alone the general public?
Currently, there are four initiatives of the cross-stakeholder global movement to make sure that the value of scholarship within society is assessed responsibly and appropriately. The four initiatives are the Leiden Manifesto, the San Francisco Declaration on Research Assessment (DORA), the Democratization of Knowledge Movement, and the Proposal to replace Journal impact factor. To date, more than 2200 organizations from more than 90 countries have signed DORA. 8 Another assessment, namely the TOP guidelines (short for Transparency and Openness Promotion), 9 has also been introduced to check the completeness of openscience principles at the journal level. Despite the strong campaign by the advocates of open science, we believe that it would be difficult to monitor the institutions that have supported these initiatives by signing up because the Leiden Manifesto and DORA are based on self-assessment. Especially in some underrepresented countries, the path to national-level research assessments has been the subject of many high-level debates. 10 We begin to address those debates by examining the relevant bibliometrics at the global level and then considering their implications for Indonesia at the national level and for the global earth sciences community at the international level.
The present study aims to ascertain whether different scientific databases give different results for different subsets of publications such as all documents, only earth-sciences documents, only open access (OA) earth-sciences documents, and only English documents but limited by the geographical context. The hypothesis is as follows: If the subsets differ significantly, then measuring the impact of a study primarily based on those differences, or rankings, as shown by the databases and as practised by many countries (including Indonesia), will be scientifically invalid and socially discriminatory.

Databases
The full data set used for all the analyses in the present study is available online. 11 The data sets were built by searching the following databases: Dimensions, Garuda, Google Scholar (GS), Korean Citation Index (KCI), Lens, Russian Science Citation Index (RSCI), Scientific Electronic Library Online (SciELO), Scopus (retrieved through Institut Teknologi Bandung, Indonesia), and the Web of Science (WoS; retrieved through UniLaSalle, France) (Supplementary Table 1).

Search strategy
In each database, we searched the documents related to earth sciences. Because the databases have different filters and search menus, the search strategy was tailored accordingly (Supplementary Table 2). For Dimensions, KCI, Lens, RSCI, Scopus, and WoS, we used the available filters and the field of science to search for documents related to earth sciences. In GS and Garuda, it is not possible to filter the documents based on the field of science; therefore, we typed in the relevant keywords. Because GS indexes most online documents regardless of language and because Garuda indexes only Indonesian journals, we decided to use the local language as a keyword in GS and Garuda to retrieve all the relevant documents published in languages other than English.
Using all the databases, we extracted the following items of information. • Total number of records: total documents and open-access documents, by year • Type of documents: articles, conference proceedings, and book chapters • Source title: title of the journal (no filtering) • Research areas: earth sciences research areas using appropriate filters in the database • Country: the country of residence of all authors, without any filtering • Funding sources: without any filtering • Language: English and languages other than English (depending on the availability of language filters)

Statistical analysis
We used descriptive statistics to describe the characteristics of documents in each database to generate the tables and then generated appropriate graphs using Datawrapper.de, an online platform.

Total number of documents and their distribution
At the international level, our searches retrieved (Table 1) almost 4.5 million earth-sciences documents from Scopus and WoS, more than 4.6 million from GS, nearly 2 million from Dimensions, and almost half a million from Lens. At the national or regional level, SciELO-indexed journals fetched more than 46,000 documents, just under half the total from the French database HAL (more than 95,000 at hal.archives-ouvertes.fr/browse/domain). The Russian Science Citation Index retrieved 39,581 documents; the Korean Citation Index (KCI) retrieved 17,156; and Garuda (Indonesian national indexer) retrieved 4027.
The number of articles published in English (OA and paywalled articles combined) as retrieved from the international databases (Scopus and WoS) -a total of more than 1 million documents -greatly exceeded the numbers retrieved from the national or regional databases: KCI, RSCI, and SciELO together yielded only about 25,000 documents in English, indicating that documents in Korean and Russian as well as in other regional languages dominate those three databases ( Table 1).
The proportion of OA papers to the total varied with the platform: the proportion of OA earth-sciences documents in the national and regional databases (Garuda and SciELO) (Mean = 100%) was far greater than that in other databases (Mean = 24%) ( Table 1). Among the international databases, Dimensions (32.7%) and Lens (31.7%) had a larger proportion of OA articles than Scopus (26.4%) did (Table 1).
Over time, the number of both OA and paywalled earth-sciences documents kept increasing, albeit very slowly, throughout 2010-2019 in all databases ( Figure 1, extracted on 23 March 2020; link to plot and dataset: https://datawrapper.dwcdn.net/CgpLO/1/) and SciELO recorded a sharp increase since 2019.

Geographical scope of databases or indexing services
We sorted the total number of earth sciences documents from each of the four databases (Lens, Dimensions, SciELO, and Scopus; Table 2) by country using the affiliations of authors and co-authors and then ranked the countries in descending order of the total output to ascertain the top nine countries in each database. The geographic coverage varied with the database: for instance, USA was ranked first in Scopus and Lens, but not in Dimensions and SciELO; China featured in all the databases; and Russia featured only in Scopus and Lens.

Regulations concerning promotion of lecturers promotion regulation
We assigned a weighting or a score to each paper as follows: 40 (the highest score) if published in an international journal with impact factor above 0.05; 30 if the impact factor of the international journal was 0.05 or below or if the paper was published as part of a proceedings volume indexed in Scopus; 25 if published in a national journal graded either 1 or 2 by SINTA (Indonesia's science and technology index); and 10 (the lowest score) if published in a national journal not indexed in SINTA. This scoring system conforms to another regulation that defines an article as 'international' if written in English or any of the other United Nations languages, reviewed by international reviewers, and published in an international journal. This criterion automatically excludes all articles written in Bahasa Indonesia, reviewed by local reviewers, and published in local journals from being considered as reputable articles-a major hurdle for Indonesian researchers who wish to publish the findings of their research in their native language.

Discussion
Averaged over the nine databases, of nearly 50 million documents indexed by those databases only about 1.6% were related to earth sciences ( Table 1). The proportion, although very low, was relatively stable from 2010 until 2019 (Figure 1) but showed some marked changes since 2019 in all the databases.
Of the total papers related to earth sciences, virtually all those in Garuda and SciELO were OA; in the four international databases (Scopus, WoS, Dimensions, and Lens), roughly 30% were OA; and the proportion of OA papers in the remaining two databases (KCI and RSCI) was roughly 15% each.
The proportion of English-language documents in the regional or national databases (KCI, RSCI, SciELO) was approximately 26%, whereas virtually all the documents (approximately 98%) in Scopus and WoS were in English.
Thus the language bias in the well-known international, and mostly commercial, databases was obvious (Table 1). These databases also showed a clear geographical bias. For example, the smaller proportion of OA earth sciences documents (Table 1) in the international databases shows that these databases have not seriously pursued equity or democratization in access to knowledge. In contrast, virtually all the papers indexed by the national or regional databases from Latin American countries (SciELO) and Indonesia (Garuda) were OA (Table 1).
Even if we consider the proportion of earth sciences documents -whether OA or paywalled -in the two commercial databases, namely Scopus and WoS, we notice the skewed geographical distribution ( Table 2): both the databases list the papers only if the journal in question is registered with them, and it is the responsibility of the journal to have it thus registered. Therefore, the coverage depends on the journal manager's perception of the reputation of Scopus and WoS as indexing services (journal managers may register their journal in either or both databases). However, in Dimensions and Lens, the data are obtained from registered DOIs (digital object identifiers) from CrossRef and not based on registration. The other four databases, namely SciELO, KCI, RSCI, and Garuda, the coverage shows a clear geographical focus, which is only to be expected given their explicit purpose.
These language and geographical biases indicate that the international databases, with their own content selection committees, have not fully appreciated the diversity of scientific output. The coverage of Scopus and WoS might be geographically selective-a critical bias that affects virtually all bibliometric analyses using those platforms. Unfortunately, those international databases dominate the criteria for assessing research as well as researchers in Indonesia and also in other underrepresented countries, which rely on quantitative measures. All those biases mean lower incentives for Indonesian scholars as well as lower visibility of documents written by them in languages other than English, which, in turn, would decrease their contribution to the local community or to their own country. In other words, the current policy adopted in Indonesia for assessing research actively discriminates against Indonesian researchers and their written work in any language other than English. 12 There is also a distorted perception by some elements of Indonesian government that the impact of research is often about intellectual property (including patents). 13 We need to prevent this one-size-fits-all paradigm from being applied to the world of research and publication, and we need to uphold the academic freedom of each institute, programme, or discipline to choose their own measures of research usefulness based on their assessment and suited to their environment. For example, many countries should in fact encourage publishing research findings in local languages by rewarding such efforts.
Apart from the simple citation indicators, Moravcsik et al. have proposed the Active Citation initiative as a way to give more context to citations; 14 however, the proposal is yet to find many takers. Active citation is a citation method that emphasizes the transparency and accountability of citations by requiring each citation to be accompanied by a narrative or annotation on how the source of the citation inspired the author to express certain thoughts. With the breakthrough in the technology of text mining that can use active citations, the willingness of international authors to read and cite local articles written in English or in local language must be strengthened. Language should not be a major barrier, 15 especially when we have created such methods as Google Translate based on natural language processing. It is indeed unfortunate that Purnell (2019) discussed the publishing situation in Indonesia 16 citing only those articles listed in WoS or Scopus and ignoring other relevant articles published in local journals by Indonesian researchers.
Another initiative is to combine such mainstream metrics as citation counts and h-index with such social-media-related criteria as download counts, total views, Altmetric score, and ImpactStory score derived from social media measures 17 The idea behind this suggestion is to avoid the overuse of quantitative mainstream metrics, especially when those are not article-level metrics. This idea, we think, might be feasible with some caution that instead of aiming at an aggregate score, we should use a multi-axis chart to present the relative position of each article along as many diverse axes (each representing one criterion) as possible.
In addition, we recommend that the impact of research be measured more meaningfully instead of resorting to mere comparison of numbers. We need to be constantly aware that not only the impact-factor bias but also the database bias have affected our systems of evaluating research for so long that the systems have lost their connect with society at large. In the field of earth sciences, we suggest that quantitative bibliometric indicators, including citations, cannot replace qualitative peer review in judging scientific impact, as suggested by the Royal Netherlands Academy of Arts and Sciences. 18 Many aspects of research cannot be measured quantitatively, especially its socio-cultural impact. 19 The interplay could be best represented by open peer reviews that provide opportunities for peers to conduct quantitative reviews to complement the main qualitative reviews, both pre-print, and post-print, such as that practised in ScienceOpen.com.
Alternative assessment platforms that value diversity and transparency at the article level, such as Redalyc.org, CurateScience. org, TOPFactor.org, and ReplicationIndex.com, should be initiated as a pilot effort by inviting several publicly funded and privately funded universities to participate. Several non-profit funding organizations, including the Sloan Foundation (Sloan.org) and Chan Zuckerberg Initiative, 20 are likely to fund this kind of initiative.
In principle, we should not rigidly differentiate academic quality and societal benefits: the negative consequences of this 'Kantian dualism' between the two are reflected in the failure of scholarly communication during the COVID-19 pandemic, a failure due to the disengagement between Indonesian government and the country's academic community. 21 The impact of research should reflect how research can contribute to policy formulation and the life of most Indonesians. The contribution of research to daily life can be shown more clearly if research is widely disseminated beyond the confines of scientific publishing, for example through Jadi Gini (So Here It Is), an Indonesian Instagram account describing the results of research in a few slides (https://www. instagram.com/_jadigini/), and The Conversation Indonesia (https://theconversation.com/id) as channels for scientists to explain their work in more accessible way. The use of less formal academic venues to explain complicated science in bite-size chunks will also enhance the reputation of researchers in the wider community. As highlighted recently by Pourret et al. in Geochemistry, even if our work is published, the job of sharing our science is not finished until we have communicate it to the wider public. 22 We wish to emphasize that we are certainly not against scholarly publication. Although we are talking about the decolonization of our scientific work from international commercial publishers and databases, we do not want to step back from the scientific life. It is understandable that some scientists -perhaps because they are tired of the colonization game being played out in the world of international scientific research and publication 23 -wish to take the radical step of abandoning the process of scientific publication altogether or to consider it less of a priority. 24 However, instantism, political correctness, and misinformation concerning the scientific field cannot be justified, because they may endanger not only the health of scholarly literature but also put human life itself at risk.
To measure the impact of research more effectively, we wish to advocate some open-science perspectives. First, at the level of the research and publication ecosystem, the diversity of research outputs and outcomes and public engagement need greater attention. Initiatives such as Jadi Gini and The Conversation Indonesia, as discussed, meet this need to some extent. Besides, by using such initiatives as CurateScience, scientific articles could be customized and used flexibly. Second, at the journal level, OA to scientific outputs and scientific processes is urgently required. In terms of the disclosure of scientific publications, we recommend that the practices advocated on Redalyc and ScienceOpen be implemented as a guide in implementing other systems parallel to those initiatives. In terms of the openness of the scientific process that promotes integrity, accountability, and reproducibility, we recommend that such initiatives as TOP Factor and Replication Index be used as guides. Third, at the article level, various initiatives, such as Active/Annotated Citation, Contextual Citation (Scite.ai), and natural language processing need to be employed to ensure accurate interpretation of the article's content. Overall, if indicators of research and publication performance need to be quantified, we recommend that multiple coordinates (based on multiple axes derived from the initiatives mentioned earlier) be used as parameters instead of a single aggregate score in deciding academic tenure, research funding, and related matters.
The limitation of this study is that in some search strategies (Supplementary Table 2), we were unable to include the Boolean operator OR in our search because of the limitations of some databases. Besides, we cannot definitively explain why KCI and RSCI showed the lowest number of indexed articles. Another limitation is that we did not check for the possibility of duplicates in the search results, although such a possibility should be kept in mind in conducting further studies.
The present study concludes that, at present, more attention is given to scientific papers written in English than in any other language, regardless of the relevance and contextual targeted audience, and, regrettably, Indonesia's national policy promotes this bias. Most of the funds have been used for gaining international acknowledgment by publishing more and more articles in internationally indexed journals, ignoring such more upstream components as strengthening the infrastructure for research and its publication.
We believe that every piece of research seeks to solve technical and social problems so as to better the lot of people as social beings. Therefore, to measure scientific performance, we need a combination of quantitative and qualitative measurements 25 to bring to the surface what lies buried below the surface. This study strongly calls for a new set of perspectives to measure the impact of research.