ISPR 2009 - International Symposium on Peer Reviewing: ISPR 2009

International Symposium on Peer Reviewing: ISPR 2009

in the context of
The 3rd International Conference on Knowledge Generation, Communication and Management: KGCM 2009

July 10th - 13th, 2009 ¨C Orlando, Florida, USA

Background, Motivation, and Purpose

In a survey of members of the Scientific Research Society, "only 8% agreed that 'peer review works well as it is'." (Chubin and Hackett, 1990; p.192).

"A recent U.S. Supreme Court decision and an analysis of the peer review system substantiate complaints about this fundamental aspect of scientific research. Far from filtering out junk science, peer review may be blocking the flow of innovation and corrupting public support of science." (Horrobin, 2001)

Empirical studies have shown that assessments made by independent reviewers of papers submitted to journals and abstracts submitted to conferences are no reproducible, i.e. agreement between reviewers is about what is expected by chance alone. Rothwell and Martyn (2000), for example, analyzed the statistical correlations among reviewers' recommendations (made to two journals and two conferences) by analysis of variance and found out that for one journal "was not significantly greater than that expected by chance" and, in general, agreement between reviewers "was little greater than would be expected by chance alone."

These are just three examples of an increasing number of facts that are indicating that more research and reflections are urgently needed on research quality assurance and, specifically, on Peer Review. "Peer Review is one of the sacred pillars of the scientific edifice" (Goodstein, 2000). "Peer Review is central to the organization of modern science¡?why not apply scientific [and engineering] methods to the peer review process" (Horrobin, 2001). Why not apply peer review to current peer reviewing methodologies.

Research and reflections on Peer Review have been mainly addressed by Bio-medical communities, and the results have been mostly shared via five International Congresses on "Peer Review in Biomedical Publication", the first of which was held in 1990. The sixth of these congresses will be held on September 2009 and is being organized by the Journal of The American Medical Association (JAMA) and the BMJ (British Medical Journal) Publishing Group.

We are convinced that reflections and research on Peer Reviewing is also needed in other scientific and engineering disciplines, as well as in multi-, inter- and trans-disciplinary research, technological projects, and Knowledge Management in Business and Government. Methodologies applied, and problems found, in peer reviewing in diverse academic disciplines, can synergistically cross-fertilize each other, and can contribute to knowledge quality assurance in the area of Knowledge Management, which would benefit the private and the public sectors and, in general, what has been called as "Knowledge Society". This is why we think that the multidisciplinary context of WMSCI 2009, and its collocated conferences, might fertilize the required cross-disciplinary opportunities.

Conceptual and methodological research and reflections on Peer Review are being increasingly desirable, important and even necessary in academic disciplines and interdisciplinary programs and projects. Peer Review is a research evaluation process which, in turn, requires to be researched and, in turn, peer-reviewed. Peer Review of Peer Review methodologies is urgently being required.

" 'Peer Review' is a name given to a principle that research should be evaluated by people bound by mutual trust and respect who are socially recognized as expert in a given field of knowledge." (Steve Fuller, 2002, Knowledge Management foundation p. 232; emphasis added) But, "peer review" is also a name given to the processes and/or the methodologies of implementing the mentioned principle and achieving the implied objective. In any case, "peer review" refers to knowledge quality control (as a principle, an end, or a mean). But the fact that only 8% of the members of the Scientific Research Society agreed that 'peer review works well as it is' means that peer review "as it is" needs to be, in turn, peer reviewed and, consequently, researched. Although we all agree on "peer review' as principle there is a solid disagreement regarding the effectiveness of the methodologies being applied into achieving the objectives implied by the commonly agreed principle. In the survey made to the members of the Scientific Research Society, 92% of its members disagreed with the actual implementations and methodologies applied in peer reviewing processes.

The almost unanimous agreement about peer reviewing as principle, and the huge disagreement about its current methods, are a clear sign that more efforts are needed in scientific and engineering research and development in order to identify more effective methodologies and support systems (especially with current Information and Communication Technologies) so the real purpose of peer review (based on its principle) is better fulfilled.

The Organizing Committee of the International Symposium on Peer Review: ISPR 2009 thinks that the multi-disciplinary approach of The 13th World Multi-Conference on Systemics, Cybernetics and Informatics: WMSCI 2009 (and its collocated conferences and symposia) and the ICT orientation of many of their participants, might be a fertile context for academics and researchers who can help, through their experience and knowledge, their reflections, research, ideas and opinions, to identify solutions, innovations and support systems for more effective peer reviewing approaches, models, and methodologies.

The Organizing Committee of ISPR 2009 invites scholars, researchers, editors, publishers, authors, readers, professionals and, in general, any user or person affected by or affecting scientific and engineering peer review to submit articles related to their research reflections, ideas, hypothesis, models, etc. on peer review and how to improve it. Among the kinds of submissions accepted are the following:

Research articles
Reflection articles
Literature research papers
Experience-based Position papers
Research proposals
Engineering Design Proposals
Decision Support Systems Engineering applied to editorial decisions.
New ICT-based peer reviewing models

References

Chubin, D. R. and Hackett E. J., 1990, Peerless Science, Peer Review and U.S. Science Policy;
New York, State University of New York Press.

Horrobin, D., 2001, "Something Rotten at the Core of Science?" Trends in Pharmacological
Sciences, Vol. 22, No. 2, February 2001. Also at http://www.whale.to/vaccine/sci.html and
http://post.queensu.ca/~forsdyke/peerrev4.htm (both pages were accessed on February 1, 2009)

Goodstein, D., 2000, "How Science Works", U.S. Federal Judiciary Reference Manual on
Evidence, pp. 66-72 (referenced in Hoorobin, 2000)

Rothwell, P. M. and Martyn, C. N., 2000, "Reproducibility of peer review in clinical neuroscience Is
agreement between reviewers any greater than would be expected by chance alone?" Brain, A
Journal of Neurology, Vol. 123, No. 9, 1964-1969, September 2000, Oxford University Press