Rejection rate and reasons for rejection after peer review: a case study of a Russian economics journal

Background: Peer review remains the only way of filtering and improving research. However, there are few studies of peer review based on the contents of review reports, because access to these reports is limited. Objectives: To measure the rejection rate and to investigate the reasons for rejection after peer-review in a specialized scientific journal. Methods: We considered the manuscripts submitted to a Russian journal, namely ‘Economy of Region’ (Rus Экономика региона), from 2016 to 2018, and analysed the double-blind review reports related to rejected submissions in qualitative and quantitative terms including descriptive statistics. Results: Of the 1653 submissions from 2016 to 2018, 324 (20%) were published, giving an average rejection rate of 80%. Content analysis of reviewer reports showed five categories of shortcomings in the manuscripts: breaches of publication ethics, mismatch with the journal’s research area, weak research reporting (a major group, which accounted for 66%of the total); lack of novelty, and design errors. We identified two major problems in the peer-review process that require editorial correction: in 36% of the cases, the authors did not send the revised version of the manuscript to the journal after receiving editorial comments and in 30% of the cases, the reviewers made contradictory recommendations. Conclusions: To obtain a more balanced evaluation from experts and to avoid paper losses the editorial team should revise the journal’s instructions to authors, its guide to reviewers, and the form of the reviewer’s report by indicating the weightings assigned to the different criteria and by describing in detail the criteria for a good paper.


Introduction
Peer review is a major mechanism for selecting significant and reliable scientific contributions from among the growing number of dubious submissions to scholarly journals. In addition, comments and recommendations from reviewers contribute to improving the reviewed contributions. 1 The increasing number of submissions encourages editors to keep refining the process of peer review, and a variety of forms of peer review (closed, open, single-blind, double-blind, 2,3 postopen-peer-reviewing, 4 or hybrid open peer review 5,6 ) aim to make the process more transparent and to improve the quality of a journal's content.
Authors select a journal for publishing their work not only for the journal's impact factor and editorial policy but also based on the journal's rejection rate. Indirectly, this indicator is believed to reflect the quality of peer review and the entire publication process. 3,7 The level of selectivity is measured by the acceptance or rejection rate. 8 In most cases, studying rejection rates can help authors in selecting the journal for publication or in estimating the average rejection rate for a particular discipline 9 or publisher 10 given the significant differences in rejection rates depending on the research area 11 and the journal's policy. 12 The average acceptance rate for journals indexed in the Web of Science and Scopus is between 35% and 40%. 13 However, there is scant research focusing on rejection, because of the lack of access to peer reviewers' reports for analysis. 14 In such cases, two sources can be sampled, namely surveys of authors of rejected manuscripts 13,16 and analyses of editors, publishers, or associates of the journal. 15,16 However, little research relies on the visible statistics of submissions and rejections.
The reasons that reviewers and editors mention in their negative reviews rarely become the focus of research. 17 The process of research evaluation varies with the academic discipline. In different fields, papers can have different structures, 18 criteria of integrity, and novelty. 19 Hesterman et al. conclude that the most common reasons for rejection by a medical journal are flaws in methodology and study design, poor reporting of the methodology, poor statistical analysis, overstatement of conclusions, problems with covariates or outcomes, and problems with the control or case group. 20 Researchers in history and the humanities appreciate the originality of research approach, whereas those in social studies value the originality of method. 21,22 The most frequent reason for rejection by economics journals is lack of significant contribution to science. 23 Usually, the editor-in-chief or the editorial team makes the decision to reject a manuscript either immediately (desk rejection) or after one or several rounds of peer review. 24 The primary reasons for desk rejection are a mismatch with the scope of the journal 6 and plagiarism. Other reasons for immediate rejection depend on the policy of the journal and include the type of paper, insufficient novelty, and poor English. 25 Desk rejections can constitute 50% of all rejections. 26 While researchers of peer reviews note the differences in peer reviewers' opinions about the same paper, 3 a qualitative analysis of such opinions is required.
To assess the robustness of the peer-review process, we measured the rejection rate and the reasons for rejection after the peer review in the case of one scientific journal, namely 'Economy of Region' .

Methods
To assess the quality of peer review, the editorial team of the 'Economy of Region' (Ekonomika Regiona, ER) agreed (1) to analyse the rejection rate for submissions from 2016 to 2018, (2) to examine the reasons for rejection, and (3) to compare, for each manuscript, the opinions of two reviewers who had reviewed it.

Manuscripts
The journal 'Economy of Region' (www.economyofregion. com) is a Russian journal indexed in the Russian Index of Science Citation (RISC), Scopus (from 2010), Emerging Science Citation Index (ESCI, from 2016), and a few other databases. The journal publishes original papers (about 90%) and reviews (about 10%) concerning regional economy in Russian and in English.
We confined our analysis to the period 2016-2018 for technical reasons. The editorial team has used the internally developed online submission system since 2015, when it was set up and regulated. This system allows reporting of the number of submissions, reviews, and editorial decisions. Manuscripts that were desk-rejected (mostly on account of a high percentage of text similarity or mismatch with the journal's scope), those not resubmitted to the journal by authors after revision, and those retracted by authors after submission were excluded from the analysis.

Reviewers' reports
The journal 'Economy of Region' follows the double-blind process for peer review, and usually sends each submission to at least two experts in the field; typically, one reviewer is internal, and the other is external. Peer reviewers complete a two-part reviewer's report form. The first part comprises a scale, from 0 to 30 points, divided as follows: problem relevance (a maximum of 3 points), clarity of formulation of hypothesis (5), novelty of research results (12), methodological and academic contribution (7), and practical application of the results (3). The higher is the score, the higher the quality of the manuscript under review. A score of 30 points means that the reviewer recommends that the paper be accepted without revision; a score from 25 to 29 means accept with minor revision; that from 20 to 25 means accept with major revision; and that from 0 to less than 20 means reject the paper. The second part comprises detailed comments by the reviewer (this was the part from which we extracted the shortcomings of the manuscript). The editorial board takes the final decision based on the recommendations of the two reviewers and the average of the scores assigned by the two reviewers.
Authors receive a decision letter containing a summary of the reviewers' comments and the editorial decision. The four possible decisions are as follows: accepted, accepted with minor revision, accepted subject to a major revision, and rejected.
First, we calculated the rejection rate index (RRI) using the following formula: RRI = (Σ Rejected manuscripts / Σ submitted manuscripts) × 100% We should note that the accuracy of this calculation is limited, because some of the manuscripts rejected in 2016 had been submitted in 2015, and some that were submitted in 2018 were peer-reviewed in 2019 and thus excluded from the sample. However, we expect that these minor deviations will have a limited impact on the rejection rate.
Review reports varied considerably in length (from a few phrases to several pages) and content. To analyse the most common shortcomings of the rejected manuscripts, we calculated the sum of all the 'yes' responses by the reviewers each year.
We also conducted a content analysis of the reviewers' detailed comments. The comments were grouped into five categories of shortcomings, as described below. 1. Text similarity. The editorial team uses iThenticate, Antiplagiat, or Rucontext similarity-detection software in two stages -at first submission and before publication -to detect unethical submissions (with high degree of similarity suggesting plagiarism). 2. Mismatch with the scope of the journal. The title, abstract, content, methods or conclusions are outside the scope of the journal. 3. Lack of novelty. The authors present a familiar and wellcovered topic. 4. Weak research reporting. This category includes a range of comments: the hypothesis is deficient or unclear; the background or methodology is weak; data have not been processed or described, are erroneous or unreliable, misinterpreted or outdated; results are unclear; results lack application; conclusions are not justified; the logic is defective (a major part of the manuscript does not correspond to the scope of the journal or to the manuscript's topic); and the references are not relevant (too few, old, local, or incomplete).

Design errors.
This category includes such shortcomings as non-academic style of writing, grammatical or factual mistakes, inaccuracies, poor quality of English or Russian, and additional material inaccuracies (figure captions and table titles missing or inconclusive or not contributing to better understanding of the manuscript). Finally, we analysed the extent of variation in reviewers' evaluation of each rejected manuscript.

Statistical analysis
For quantitative assessment of the scores assigned by reviewers, we calculated the descriptive statistics using Stata/MP ver. 13.0. The scores (from 0 to 30) and the number of shortcomings were analysed.

Rejection rate
A total of 1653 manuscripts were submitted to ER from 2016 to 2018: 543 (33%) in 2016, 513 (31%) in 2017, and 597 (36%) in 2018 (Table 1). Of the total, 595 (36%) manuscripts were either desk-rejected, not resubmitted, or withdrawn by authors before peer review and 734 (44%) manuscripts were rejected after the peer review, giving an average rejection rate of 80% (Table1). The descriptive statistics of the scores assigned to the rejected manuscripts show the mean score for the rejected manuscripts to be 13.05 out of 30 points and the median value to be 14.0 (Table 2). These values correspond to the defined score for acceptance, which is from 20 to 30 points. The mean and median values of the scores for specific time-series show a slight increase, as do the mean and median values for the shortcomings: on average, each report contained 4.5 comments related to the shortcomings of each rejected manuscript.  We analysed the statistics of reviews grouped by the number of authors of each submitted manuscript and the reviewers' scores ( Table 3). The mean score decreased as the number of authors increased, implying that the quality of manuscripts with more than three authors was lower that of the manuscripts with three or fewer authors.

Reviewer-defined shortcomings of rejected papers
Of the total, 66% of the shortcomings were related to the quality of reporting (Table 4). Over the period 2016-2018, the most numerous in this category of shortcomings was insufficient justification for conclusions (13.5%), followed, in that order, by weak methodology (8%), weak formulation of hypothesis (7.7%), weak theory (5.9%), and irrelevant references (4.8%). The second category of shortcomings was related to mismatch with the journal's research area (15%). These were subdivided into cases of titles not corresponding to the journal's research area (10%), the content being outside the scope of the journal (4%), or major part of the manuscript not corresponding to the topic or to the scope of the journal (1%). In rare cases, this became clear during the moderation process (in reading the titles of the manuscripts), and such manuscripts were desk-rejected. More generally, reviewers can pinpoint the mismatch between the manuscript and the journal's academic focus. The most common case was of papers on economics that had simply added the word 'region' or 'regional' to the title or to abstract, although the manuscript clearly did not consider any issues related to regional economy.
A few shortcomings were related to lack of novelty (10%), design errors (8%), and text similarity or plagiarism (1%). A total of 38 cases of authors' misconduct were identified after the peer review through the use of similarity-detection software packages such as iThenticate, Antiplagiat, or Rucontext. The majority of these submissions contained self-plagiarized fragments, although a few of them were clear cases of plagiarism.

Agreement between reviewers
We considered the cases in which the scores assigned by the two reviewers to the same manuscript differed by more than 20%. In 69% of cases, the reviewers were unanimous in their recommendation to reject a manuscript. The number of submissions on which the two reviewers did not agree was lower in 2018 (26%) than that in 2017 (38%) and 2016 (34%). Usually, in these cases the editorial team sends the manuscript to a third expert, although this delays the final decision on the submission.

Discussion
A rejection rate of 80% shows a sufficiently rigorous process of selection. Of the total rejections, 45% were related to rejections after a negative peer review and were analysed in the present study. Since a score below 20 corresponds to rejection, the mean score of manuscripts analysed here was from 0 to 20. The most frequent reasons for rejection were weak conclusions and methods, unclear hypothesis, and mismatch between the title of the manuscript and the journal's research area. At the same time, the quality of the design and the style of manuscripts had improved in 2018 (6.7%) compared to that in 2016 (9.7%) ( Table 4). Between 2016 and 2018, the editorial board had twice revised the journal's instructions to authors on the journal's website. The board had made the moderation (first checking) of submissions more detailed and developed interactions with authors to explain the necessity of revising the manuscripts to match international publication standards. Such shortcomings as poor referencing, old data, and lack of practical applications of the results are of less importance to peer reviewers, and comments related to the relevance of references accounted for less than 5% of the total. However, these criteria are crucially important to ER as an economics journal, indexed in international databases. Under conditions of economic crises, research based on old data can result in erroneous conclusions and irrelevant recommendations. Practical applications of research results are also significant for the regional economy, which is the focus of the journal.
The content analysis showed that the most frequent comments made by the reviewers did not correspond to the editorial vision of the most significant shortcomings. Moreover, the reviewers are to better explain some of the deficiencies. As the reviewer report form allows free comments, the reviewers can hardly define the two first shortcomings and sometimes spell them without explanation.
Content analysis of the reviewers' reports revealed two serious problems. First, not all reviewers pay attention to the most important aspects of the manuscripts: some reviewers apply more formal criteria; some appreciate the logic behind the manuscript's content, and yet others look for the essential originality. Second, all reviews are to some extent subjective.
Although in 69% of the cases the reviewers had been unanimous in their recommendation to reject a manuscript, in nearly a third (32%) of the cases the reviewers' recommendations were contradictory. This demonstrates the reviewers' subjectivity and lack of agreement on what is crucially important for the journal. Earlier research has shown that whereas reviewers mostly agree on accepting a submission, more frequently they differ in their opinions when suggesting revisions or when rejecting a submission. 27,28 As disagreements between reviewers delay editorial decisions, 29 the review process needs to be improved.
As a solution to the problems identified, the reviewer's report form should be improved so that it contains wider criteria and detailed descriptions of each. The weighting assigned to each criterion should reflect the editorial vision of the relative importance of the criteria of a good paper as seen in the light of the scope of the journal. More specifically, the report should include the relevance of data and of references among the criteria for evaluating manuscripts. The editors can send appropriate instructions for completing the reviewer's report together with the manuscript to be reviewed. Furthermore, the editorial team should constantly seek to expand the strength and the geography of the pool of reviewers to reduce reviewer subjectivity. A guide for reviewers available on the journal's website and online training for reviewers will also contribute to deeper understanding of the reviewer's role.
In addition to these measures, the journal also needs to keep developing its editorial infrastructure including appropriate software, website design, author guidance, information support, and training of authors.
Two limitations of the present study should also be noted. First, we had no statistical data on the types of papers. The journal mostly publishes original papers; review papers are not as frequent. However, we did not consider the type of paper in analysing the reasons for rejection. Secondly, the number of peer reviewers per manuscript was not taken into account: although most submissions are sent to two peer reviewers, the editorial board enlists a third and even a fourth expert if the first two reviewers send conflicting recommendations.
Our findings can be used by the editors of all journals that are indexed in international databases to improve the process of review and will also help both authors and reviewers by extending their understanding of the criteria and the stages of the review process.