Recently I was asked why academics (or at least tenure committees) respect peer-reviewed publications over other high-quality articles published in non-peer reviewed spaces (journals, newsletters, blogs, wikis, etc.). I have been thinking about this quite a bit over the last couple years, considering that I have a career ahead of me (hopefully) involving tenure review. However, I neither believe that traditional "high-quality" journals should have the power that they do over evaluation of academic prowess nor do I believe that they are the best place way to disseminate information. So, which direction do I go?
As journal subscribers, we count on editors to censor to an extent. We want articles that are on topic (as per the focus of the journal) as well as articles that meet generally agreed upon standards as far as method and presentation. These are generally accepted based upon the readership (they vote with their subscriptions). Publication/submission requirements change over time as the readership changes. This filtering for content and format results in publications that are easier to read in that readers know what to expect in both content and form. I like this. It allows me to quickly process many articles in a short amount of time
However, I don't agree that we should judge the value of an article by the journal in which it is published. While, I see this as a natural tendency for humans to identify with the familiar, I think that we need to be broken from this habit (or safe zone). For example, if I see that an article was published in TESOL Quarterly, I know that, in my experience, articles in this journal are generally of high quality. So, I'll assume that this one will meet my standards for quality. Whether it does or not will have to wait until I actually read it. This is where the real problem occurs. Most people will assume that the article is of high quality because it appears in the journal. Most often, they simply don't know how to judge high vs low quality (a somewhat subjective judgment) therefore they leave the decision up to the editors/reviewers. I would include many faculty in this criticism. They point to articles in specific journals when trying to strengthen their arguments that aren't much more than editorials. They take these statements of opinion as fact and pass it on to their students without disclaimer (or training) for them to make up their own minds.
So, now we are stuck in a vicious circle. Academics are required to publish in these journals and they are judged by the "quality" of the journal in which they publish. Those who don't want to go in this direction are driven out of the field, thus only those who tow the line remain.
I'd love to see an academic version of Digg (http://digg.com) in place of refereed journals. Let those in the field (everyone in the field) determine the quality of a piece of writing and take in out of the hands of a few. This would get research out to the public faster and with less editorial interference than the current system.
Add to this the ability to create collaborative documents that can be edited beyond publication and you get a system of ranking and improving on publications that goes far beyond the current , slow, closed system.
What do you think?
Mr. Craig, Your stance is reasoned and sensible. I think that all too often members of any given professional community feel threatened by the unknown and untested. For academics to feel this way, however, is tantamount to fearing the very core of what learning is all about.
ReplyDeleteI'd not seen Digg before, having taken a look at it today I see that it appears to be a moderated news aggregate site that, as you said, allows some form of "peer" commentary. Your pointing to this site calls to question another important term in this discussion. What is a "peer." Are the only viable judges of academic work those who hold positions and titles accepted by academic circles. What is the meaning of tenure? What is the meaning of a PhD? And how might all of the publications written and submitted to any variety of sources by all non-academics be seen as equally valuable in academic dialogue?
I think that your suggestion that one must simply regard the work in its own light--sometimes guided by the reputation of a journal or any source--and recognize that quality is not the domain of any single profession, person, or place. Thanks for your comments!
Mary
So formal..."Mr. Craig" :)
ReplyDelete"Academic" publications will not be immune from the User Created Content phenomenon that we are seeing now in all media. When anyone can publish, anyone can be an author.
Will this make is more difficult to find high-quality research? At first it likely will. However, as the movement matures, we'll naturally have people, organizations, and communities that filter the content for us. This has and is happening in all other areas of user created content.
A question that you ask is "...how might all of the publications written by and submitted to any variety of sources by all non-academics be seen as equally valuable in academic dialogue?" I would say that ideally, in this system of public filtering, quality would matter more that academic credentials. I don't really believe this, unfortunately. Quality is measured in large part by how many people cite you in published articles (as well as the ranking of those journals). For whatever reason, authors tend to cite their departmental/university colleagues. This could be a "scratch my back I'll scratch yours" arrangement, it could just be collegial courtesy, it could be that you want to boost your buddy's ratings. Whatever the reason, it's happening.
At this point in the evening I'm beginning to ramble a little too much, so I'll stop after this final comment.
The issue of publication and the dissemination of ideas in education (as well as other fields) is something that many people don't even question, much less think is a problem. This, to me, means that we are a long way off from changing the culture of the academy.
Dan
Openness of publication and access can make results rather messy -not to say overwhelming. Spotting good quality is becoming a complex task.
ReplyDeleteFor the time-being, an article saved 140 times in del.icio.us does not speak about its potential academic value. It just states that 140 people have read it.
Perhaps sites like diigo, with shared annotated bookmarks and the possibility to trace the note writer to its webpage, could open room to find diamonds in the rough outside the recognised journals in any given field. Yahoo pipes also looks promising as you mentioned in another post.
Still, as much as technology may simplify the finding process, much remains to be done about learning what makes a contribution valuable. To become more autonomous readers and not just wait until authorities decide what should be read.
Academy culture tends to change slowly. Authority and stability are at the heart of its existence. Should universities become more flexible, open, change altogether? Hard to predict their future.
Serendipitous finding. I had long added you to my del.icio.us network. I spotted you today at the FOA attendr map and I landed on this interesting conversation.
Thank you for sharing, Craig.
Thanks for stopping by Claudia.
ReplyDeleteYes, I agree that finding good research is a problem. I'd also argue that is a problem even in popular publications :)
Academic programs these days have reading lists, either implied or official. These are the readings that they find central to their field. The more current readings are filled in by the individual professors in the department. This is a recommendation system.
Most professors do rely on the top journals in the field (and students' research) to fill in their knowledge of the state of the field. This would have to change, but not much. How far a leap is it to trust one source over another. It just has to be shown that a system can adequately promote better research.
Of course, I don't want to minimize the effort in developing such a system. It would be quite a task, but it is certainly possible.
Lastly, I have to say that we must train academics to judge the quality of research better than we do today. Blind reliance on the journals has made us lazy. How often do you hear, "but it was published in..."?
Hope to see you back here again in the future.
Dan
p.s., I post a lot more in my CALL Class blog at http://iucall.blogspot.com. Have a look.