Privileged Peer Review – whose opinion counts?

While in Lisbon earlier this month, I was approached by a Masters student who expressed skepticism expressed by her review panel on connectivism. Our conversation was short, and the situation likely more complex than expressed during the brief exchange, but she stated that her panel felt that the theory of connectivism and the book Knowing Knowledge had not been subject to peer review.
I’ve heard this complaint several times – generally from students in Masters or PhD level programs. While I sympathize with the challenges of these students, I’m not entirely inclined to pursue (exclusively) publishing in peer review journals (though I have succumbed somewhat recently, with several articles in review – should be out in 2008 or so :) ). Peer review plays an important role – it is intended to provide expert critical review of concepts and ideas to ensure quality and accuracy. I’m all for that. My primary concern rests with “privilege only” accepted view of peer review. The progressive advancement of educational attainment (see OECD’s Education at a Glance 2006) indicates a society increasingly capable of engaging in complex dialogue. The throne of knowledge is now a seat available to many of society’s members. As such, it’s reasonable to assume that the opinions of even those peripherally engaged in a discipline can provide insight and value. I appreciate experts, excellence, and established processes. But I despair when the processes of validation inhibit, rather than advance, thinking and idea sharing in a discipline.
A few additional concerns:
1) Peer review is by no means a singular process of determining and ensuring quality. Implementation and context of use ultimately determine the viability of ideas. Einstein’s famous thought experiments created an impact by opening conceptual. Ultimately, even after peer review, the ideas were tested in the real situations and, over a period of decades, were amended, disproved, or upheld. Often, support came from related fields – initial analysis of Einstein’s general theory of relativity resulted in acceptance from peers in that it addressed (among other things) the problem of perihelion shift of Mercury. Experimental support was not forthcoming until much later.
2) Peer review is based on the assumption that a handful of experts are capable of validating ideas. In reality, even experts get it wrong. We all do sometimes. Opening the review process to the broadest possible number of responses, increases the possibility of quality evaluations, even though the increased commentary may appear to cloud the situation with clutter. Good ideas, however, have a way of getting connected and noticed. See swarm theory for how the aggregated activities of many singular agents can address astonishingly complex problems. Some sites, like Nature have opted to embrace open peer review. In a previous article – Scholarship in an Age of Participation – I’ve called for a model of scholarship which honors expert peer review, yet opens the door to greater involvement from the larger community.
3) Peer review time lines in traditional journals can be lengthy – months, even years. By the time the article hits the pages of the journal, the research may be outdated. While some fields develop more slowly than those impacted by technology, educational technology in particular requires a shorter lead time. Instead of relying solely on expert validation, why not permit the minds of many to grind the edges from initially posited ideas.
4) The high-privilege of the author. Authors are like lecturers – one way broadcast of information. This can be quite delightful. But the two-way flow of many developments online can be a much more rewarding experience for both the author and the audience. PloSONE allows for collaborative review and annotation (though glancing through most articles, these features are not used extensively). When writer and audience, author and reviewer, are engaged in dialogue, more fully developed ideas and more integrated concepts are the reasonable outcome.
I did a quick search on connectivism. First page of results are as to be expected: elearnspace, connectivism.ca, downes.ca, LTC online connectivism conference, etc. By the second page (results may vary based on which country location the search is conducted), we encounter Plon Verhagen’s review of connectivism. He didn’t like it. By page 4, Bill Kerr expresses his concerns with the theory. Continue on with the search, and you’ll encounter numerous expressions of support, uncertainty, and disagreement. The reactions come from teachers, professors, researchers, business people, theorists, conference proceedings and so on. Is that peer review? Or is peer review only what happens when it is under a privileged process?
The problem with peer review does not rest with connectivism (or any other such concept expressed in non-traditional means). Connectivism has been extensively peer reviewed. I would suggest much more so than most academic articles. In the process of review, others have built on, extended, and formed the theoretical basis of connectivism (see Stephen Downes’ paper on Connective Knowledge for an epistemological basis of connectivism. He may not have intended it as such, but I see certainly many points of compatibility…and a few for debate :) ).
The real problem rests with any closed method that values a singular process too highly. Take 10 minutes. Use Google (or if particularly concerned, try Google Scholar). Research the ideas expressed. Search the authors of critiques and associated articles. The world has changed. The devices (books, paper journals) and processes (closed peer review) have given way to an overwhelming array of options. A grad student has a perspective worth considering. So does someone from related fields. In my view: throw open the doors which permit dialogue. Move foundational elements of scholarship into the hands of everyone who chooses to be engaged (”citizen scholars” as stated in this report (.pdf) by the Council of Graduate Schools. Insular evaluations are not as effective as commentary from numerous perspectives. The experts still have a role (obviously). But not as privileged as previously. In this mode, an expert carries authority in dialogue based on continual contributions and reputation. Peer review, in its current conception, appears to be more concerned with preserving privilege than with advancing knowledge. I find more zeal, energy, and creativity in the chaotic edublog community than I find in formal journals. And I personally would rather stand with others on the fertile verge of technology and education defined by enthusiasm and a spirit of advancement than to pursue a method of knowledge validation antiquated for our scholarly needs today.
UPDATE: As luck would have it, I just came across two papers on peer review and academic publishing in Inside HigherEd Articles: Ideas to Shake Up Publishing and Is Peer Review in Peril?

8 Responses to “Privileged Peer Review – whose opinion counts?”

  1. George, I have been writing about similar things (evidence, expertise, authenticity, and transparency) on my blog recently. Two of these posts within links to others are http://www.silenceandvoice.com/2007/07/scholarly_authority_and_expert.html and http://www.silenceandvoice.com/2007/07/the_transparency_manifesto_joh_1.html .
    With The Chronicle of Higher Education and its recent article on how this will be changed post Web 2.0 and the early work in The Transparency Manifesto, this area of what constitutes expertice and how it can be reviewed will surely grow in debate.

  2. Helen Nicol says:

    Having referenced your book throughout my dissertation, no doubt I will suffer for using what I consider to be a valid and relevant source, in the same way I have suffered for sticking to my guns and using Wikipedia definitions of new terms widely used in the blogosphere, the subject of my work, but not “academically acceptable.” Pompous is one word I’d use, but outdated and misguided are probably more applicable terms to associate with the problem of peer review. So strongly do I feel about the short sightedness of sticking resolutely to the concept of peer review, I nicked a large part of your post and stuck it in my own blog post Wikipedia, community self-regulation and academic peer review
    Behind you all the way with this one…

  3. Hi Jeffrey – thanks for the links. I will review them. As listed in the post, Inside HigherEd reported on two recently released reports which provide additional insight into the problems and challenges facing traditional peer review: University Publishing in a Digital Age: http://www.ithaka.org/strategic-services/university-publishing and Is Peer Review in Decline: http://papers.nber.org/papers/W13272 .
    George

  4. Hi Helen – thanks for your comment. The “validation” aspect of peer review is obviously important and I certainly don’t object to having ideas discussed/validated/critiqued. As mentioned in the post, the real problem is one of academics not seeing the value of methods beyond expert-only. All types of peer review can fail – experts miss blunders, networks get caught in a cycle of perpetuating mistakes. Neither are infallible. But taken together, with context of use, each provides the other with value.

  5. Lanny Arvan says:

    George – a few months ago you made a post on future hope against today’s reality
    http://connectivism.ca/blog/2007/03/future_hope_against_todays_rea_1.html
    where you noted, correctly in my view, that we’re way ahead of the faculty we support in terms of a vision of how technology can improve the learning. I wonder if you might pause for some reflection of how your view of peer review fits into what you said in that earlier post. Also, I wonder if you might reconsider some of what you wrote above in light of the following:
    (a) Is any of your opinion tempered by discipline? In my field, economics, there are some well known economists who both publish like crazy in referreed journals and who blog as well. The nature of their work changes in these separate venues. Is online learning different from economics in that respect?
    (b) On your point (3) there are lags in review and lags in publication and those are not the same.
    (c) Working papers are typically distributed before peer review and can exist after publication of a reviewed piece, for example in an Institutional Repository. One does not preclude the other.
    (d) On point (2) about the experts, and again I’ll turn to the economics discipline as example, a lot of this is the language in which argument is presented as well as the formal modeling and data analysis. As an author reaching to a broad audience, I’d write one way. Writing to experts, I’d write differently and the issues at stake would be different as well. I think that most disciplines require some broad type of writing, and on that point I’m most in accord with what you say. But I wouldn’t want to preclude the more narrow, high-entry-requirements-for-the-reader type of writing. And for that I’d prefer peer review. Consider the proof of Fermat’s Last Theorem. Wouldn’t you want that proof to be judged by a narrow community of experts? So isn’t the issue what class of arguments should be so judged rather than not to have that type of evaluation at all?
    We operate in a practitioner driven field and there “experts” may be more removed from new knowledge than the practitioners. That is its own problem. But I don’t believe it is a problem that troubles all fields of inquiry.

  6. Hi Lanny,
    The issue – like most edtech discussions – is contextual and subject to gradients instead of points in isolation. So, yes, I agree that different disciplines have different needs for peer review. As mentioned in the post, economics is one of those fields that is finding a movement (by experts) away from peer review. This phenomenon is quite different from the one I presented – i.e. individuals (not experts necessarily) publishing in informal venues, but still receiving the value of peer review from their colleagues. Experts who publish papers on their own site are possibly skirting the peer review issue all together. My argument is not to abandon peer review, but instead to question “privileged peer review”. I think it’s very important that ideas are critically reviewed. My appeal to acknowledging the views and evaluations of non-expert peers is intended to strengthen the quality of peer review by diversifying it.
    And yes, there are fields where expert only review makes sense. Mathematics, physics, medicine, etc require an established technical base or understanding. Evaluating new formulas, procedures, etc. requires an existing base. Even then I would suggest that peer review – outside of closed expert review – in community settings would be valuable. In a sense, I’m suggesting continual review in networked community spaces, rather than one-time privileged review in advance of publication. Then there are “aggregate disciplines” in which many people have opinions – such as teaching with technology. Technology, epistemology, psychology, pedagogy, and other fields blend to comprise the field. As such, the potential peer base is much larger. All this to simply say, yes, the nature of a discipline influences the potential review base.
    George

  7. Cherylynn Desjardins says:

    George,
    I am currently taking a Master of Science (Instructional Technology) on-line degree program from Ellis College (NYIT) and found your site while looking for a learning theory for distance education.
    Using “Advanced Methods in Distance Education: Applications and Practices for Educators, Administrators and Learners” (Dooley, 2005) as our text, this week’s reflection assignment discusses behaviourism, cognitivism and constructivism and asks -’Should we have a theory on instruction and learning for distance education?’
    Reading these learning theories and dated references I kept thinking surely someone has already thought about this – surely in 2007 we are researching and proposing ideas for learning in the knowledge era, the digital age and the fact that do peole really need to know anything or is it more important that they know where to find what they need when they need it.
    Connectivism theory will be cited in this week’s reflection as – ‘yes we need a theory on instruction and learning for distance education, and here it is’.
    THANK YOU!
    Cherylynn

  8. Personally I think peer review is much over rated !. Who says academic process is faultless ? . Perhaps blind peer review is the most reliable all other methods tend to be very biased due to personal relationships, networking etc.