Wednesday, March 28, 2007
Idea for Discussion: An Academic Blog Review
This is a variant of an idea John Holbo has mentioned a few times over the years. But while (as I recall), Holbo has posited blogging as a radical alternative to the old peer-reviewed journal system, I’ve been thinking there might be a need to have a system that is more formal than “whatever, it’s all good, do your thing” academic blogging, but which nevertheless preserves some of the fun and liberatory aspects of writing and publishing on the internet.
The idea came to me as I’ve begun preparing a tenure file at my current university, acutely aware that my blog writing cannot be considered “peer-reviewed” publication by any current standard. Even the rewards of occasional Boing-Boing-ish popularity (my post on “Early Bengali Science Fiction” from awhile ago, for instance) do not help, since that is really popularity rather than review. But why not institute a review of some sort? This was also, incidentally, a question raised at the MLA by Gwynn DuJardin: how can blogs be made to “count” as part of the academic process? It came out awkwardly at the time, and at the panel where it was first asked, Michael Berube swatted it down pretty summarily: they don’t count. Yes, but couldn’t they, if there were some kind of evaluation process?
My idea is to have a system of academic blog reviewing, where people self-select individual blog posts they’ve written for review by others, perhaps using a combination of Technorati tags and emailed links. The reviewers could consist of fellow bloggers (credentials no bar) as well as non-blogging academics in a given discipline, who would pubish their reviews on a central site. The reviewers could choose to be “onymous” or pseudonymous (as long as it is a consistent pseudonym, and contact information is available to site admins), and be asked to write a significant evaluation to the post in question (say, 250 words). Other reviewers and readers of the reviews could also evaluate the reviewers’ comments, as a way of maintaining standards for reviewers. Troll-like, unfair reviews would be deleted, and their authors denied reviewing privileges.
Reviewers aren’t that different from commenters in the current blog architecture, but their purpose in writing in my system is primarily evaluative—the goal isn’t necessarily to have a conversation. Just as in formal journals, reviewers can recommend revisions or corrections. However, in contrast to formal academic journals, the reviewer doesn’t recommend publication or rejection—since what they’re evaluating has already been published on a blog. Instead, they might recommend readers to check out the post in question—as in a “Digg.”
Though for many people the idea of a formal-ish evaluative process for blog writing will sound really depressing (or just boring), for those of us who are interested in blogging as an extension of our academic projects and research it could actually be pretty helpful. I want to stress that the idea here isn’t so much that people will stop writing bloggily, or only be able to submit 5000+ word posts for review; indeed, one profitable way to think of an academic blog review is as a space to “publish” shorter work, or things that cross disciplinary boundaries, or that might just be a promising riff on something. Another hope is that blog writing, even under a review process, would remain jargon-free; people would continue to presume that their readers are essentially educated lay-readers—not narrow period specialists. (That last stipulation might well be debatable.)
Even with review, it’s unlikely that one would ever put blog posts down under “peer-reviewed publications” on one’s CV. But perhaps academics could soon introduce a new category: “Peer-reviewed Writing Published on the Internet.”
I’m initially interested in a blog post review system for literary scholarship, but there’s no reason why other disciplines couldn’t also work with it. Indeed, the larger the range of disciplines included, the more likely it is to be accepted by the system as a whole.
This is admittedly just a preliminary sketch. What do you think? Is it workable? What are some of the problems you see with this idea? Are there ways to make it better?
Doesn’t “Peer-reviewed Writing Published on the Internet” already exist in the form of accredited electronic journals? Isn’t peer review an armature of the apparatus to ensure that publication is warranted, while blogging and its ilk are inherently self-publication? That the latter pre-empts the former doesn’t support a parallel process, nor would such fix the inordinate weight the system puts upon formal publication.
Nnyhav, I’m not sure I understand your last sentence. In a blog review the emphasis is on recommendations to read or not read, rather than on recommendation to publish or not publish.
A parallel process might not fix the weight the sytem currently puts on formal publication, but it would be a start. It’s a step beyond people just saying ‘great post, man’, or linking to you. And my idea here was to incorporate the architecture of blogging into the review process itself—*not* to start another online journal.
Anyway, it was just an idea. From the lack of response, I gather The Valve is not overflowing with excitement!
People don’t usually get real excited about building another process for peer review, or anything else for that matter. Dreaming of such a thing is easy, doing it is, well, doing it. What you’ve done here is most of the heavy lifting, Amardeep, for which you should be thanked, whether people want to get in and do it or not. So thanks.
Here’s my one constructive criticism: peer review is assured. That is, an article goes to a p.r. journal, it gets reviewed. In the context of blogging, reviews would be voluntary, so there would be no guarantee that a post would be reviewed. So, problem.
One might think that those of us interested in finding a way to “count” our blog writing would feel compelled by self-interest to review these selected posts, as that would up the odds one’s own post/s would be reviewed. But, that’s an iffy bet because it requires a commitment of time and energy that “pays off” for the reviewer only indirectly. We blog, and blog here, partly because there’s an almost certain chance of being read and commented to; where the review site would be “extra work” and “not entertaining” in some direct way.
And you know people and delayed or uncertain gratification.
Soooo, there would need to be some kind of community commitment to review posts. Maybe reviewers could be recruited for given periods of time (say one semester) and asked to review X number of posts. A reviewer who does not meet this promise (at their own fault) would not be asked back to review later, possibly not allowed to post for review for X number of months. And, being a reviewer would be listed on CVs as well, so that the quality of one’s reviews and the good-citizenship of offering one’s acumen to the process would also be ‘rewarded’.
Then we get more incentive, both positive and negative. I suggest it because we often post to our blogs because we have something we want to say, noodle with, vent, parody, etc., but we don’t always comment because our obsessions and interests of the moment don’t always coincide with those of a post. And there’s a whole damn lot to read.
Reviewers would have to remember that one of the conventions of the blog post is that it is often a nascent idea, a partial gesture, a dipping of the toe, even an idle. And style would count. We like blogs and blogging, I think, partly because we can and are expected to write with some verve, swerve, and elan (whether joyful or cranky). So, these would become factors in the review.
It’s a good, solid start at legitimizing the work we do here on its own terms (as much as that’s possible).
Hmmm, maybe there could even be a review template. List the conventions of the academic blog post (which are fairly clear?), with numbers for rating, and then have a space for about 250 words of review. Hmmmm. Maybe. That might require a “committee”... or several as the conventions might vary between disciplines? I’ve not been reading blogs in the hard sciences....
Anyhow, good and generous thinking there, A.
Thanks Amardeep. I think something of the sort is a good idea, although admittedly excitement seems lacking at the moment. (I know I’ve been buried for days and not really doing much to liven up the place.) I’ll respond more a bit later. The short answer is that I think something of the sort could work, but it depends on coming up with something that adds some fluid intellectual functionality that electronic journals may lack. Build something that can allow people to get good critical feedback, as our book events do. The thing to do is think first about where you can build in functionality that isn’t provided elsewhere. Then think about how that can be leverage into an argument for accrediting the thing.
Aren’t blog carnivals and other aggregations already quite close to this idea of peer-reviewing blog posts? The 1st Science Anthology had some sort of peer reviewing. It would be interesting to set up a blog carnival that has a limitation in number of accepted submissions per issue. The submissions would be handled by the editor (as usual) but with the help of a peer review system (as was the case for the Science Anthology). In the end this would be very similar to a community journal but for blog posts.
I share the feeling that blogs should be more a part of the scientific communication process but I am not sure this would be the way to go. I think in these past 2-3 years we have seen on one side some blog posts coming closer to what you might expect to see in a peer reviewed journal but also on the other side, there are many more peer reviewed journals, with different degrees of expected impact or depth of work from a publication. In the middle I would place the pre-print archives that serve very well the role of freezing an idea in a indexable published document that can then be submitted to a peer reviewed journal. So the way I feel a blogs fits in science publishing would be to as the initial starting point were maybe the main result and discussion would start (in one or many posts). This could be shaped with the help of the comments into a frozen pre-print manuscript and submitted to a peer-review journal. Maybe this system of requesting reviews could be triggered once a paper reaches a pre-print server. I think this was already suggested for the arXive but I am not sure. If any review was attached to the pre-print this could be submitted alongside the paper to the journal.
Simone, thanks so much for some very helpful points.
I think the biggest problem right now is convincing people that reviewing other work will be rewarding. The best way to do that might be the guarantee that if you write reviews, your own work is more likely to be reviewed.
Another point you make, about whether there is in fact a set of conventions for the academic blog post, is one I’ve been thinking about since putting up this post yesterday. What “academic blogging” is has been talked about some by other Valve people (esp. Scott Eric Kaufman). In order for this model to work, a broader definition of “academic blogging” might have to be worked through—probably collectively.
Alternatively, if there isn’t a single set of conventions that everyone agrees on, people who want to be reviewers could be asked to write and publish a vision statement of some kind: this is what I think a good academic blog post in [literary studies or another discipline] might look like. So even if not everyone agrees on a standard, people who write reviews will be asked to give a framework for what at least *their* standards are.
Pedro, thanks—I’ll have to check out the science anthology you mention. I think the idea I’m proposing here might be the next generation of the blog carnivals, and the difference is precisely the one you mention: there will be editors and reviewers, and an evaluative process.
I think the sciences are well ahead of literature when it comes to disseminating published work—we don’t have anything like arXive either at present.
John, I’d be curious to know more about what you mean by “fluid intellectual functionality.” To me, that sounds like the ability to make comments? As I’ve been thinking a little about what an academic blog review would look like technically, I’ve been thinking that initially it might actually be pretty bloggy—running on Moveable Type—with the possibility that people could write comments on reviews.
But I agree, accreditation would have to come later.
Up to a point, I really like this idea: you are talking about a central academic blog switchboard that would be a blog about other academic blog entries, as well as (hopefully) a great library of links (this is one case where somebody would be truly obliged to make the blogroll comprehensive and up-to-date).
This would resemble a perpetual Academic Blogging Carnival, but it would have significant advantages: it would stay at the same URL and would provide reviews rather than merely sifting links.
Here are three problems I see ahead:
1. Lack of reward
Most bloggers are looking for more readers: they want more friends, better professional contacts, and a reason to get out of bed in the morning. When they review another person’s blog or post, they stand to gain. Either they will make a friend out of the blogger by reviewing them positively, or they will make friends with that blogger’s enemies by writing a good roast.
A reviewer, however, does not stand to gain as much, both because the style will be more constrained, and because the review blog isn’t their own personal URL. This is particularly the case with negative reviews.
2. What Gets Reviewed?
Just as important as making judgements about a particular post is making judgements about an entire blog—whether, over the course of many months and many different subjects, it remains a worthwhile read. There’s no harm in sending readers to an uneven or infrequently updated site for the sake of a good post, of course. Still, such reviews are limited by their inability to capture the whole living document in which a given post is suspended.
This is my most serious concern. Blogs have to be able to differentiate themselves from academic journal writing and academic e-mail lists, and I worry that an academic blog review site will further the general impression that academic blogs are joyless back channels for specialists.
Readers are capricious when it comes to the blogosphere. They want serious content, but they also want “fun,” and they want the same sites to be able to provide both. An academic blog review, with its connotations of professional respectability, risks making blogs more conservative (not politically, but methodologically). If that happened, readers might become apathetic or scarce, and still nobody would be actually getting paid.
My approach to academic journal articles is entirely grudging: I read what I have to when I’m researching a paper, quote a few lines, and forget the article forever. I don’t want to see that happen to the blogosphere, where my level of engagement is much greater.
Joseph, your points 1 and 3 are pretty much things I’d already kind of accepted as serious problems with my system as is.
The only good reward I’ve been able to think of is that people who do reviews earn goodwill from readers and other reviewers, which might make their own work more likely to be reviewed. (Maybe) I have to admit, the hope of reciprocity probably isn’t enough to keep things going. Without a critical mass of participants, the process could very quickly shrivel up from indifference.
(Since you mention payment, one way around it might be to get a start-up grant, and then be able to pay reviewers a small fee until things get going. The start up grant would also pay for someone to code in some special features to the site, which will go beyond what blog software can usually do. And yes, this is strictly pie in the sky...)
On standardization, I’m dreaming of some kind of sweet spot between intellectual/scholarly value and the first-person, informal style of blog writing. I know that what that means will vary from person to person, so that’s why I was imagining (in an earlier comment) an arrangement where the reviewer is asked to produce her or his own sense of what the criteria should be. That is admittedly a ‘weak’ standard (though weakness is another way of saying flexibility), which may or may not work. But perhaps if everyone has a sort of understanding that part of what is being evaluated is how entertaining and interesting the writing is, people will feel encouraged to take risks and play.
On reviewing whole blogs vs. individual posts. I see the value of reviewing whole blogs, but I gravitated towards individual posts because it can often be really difficult to compare or even read whole blogs, especially since most of us publish a fair amount of stuff on personal blogs that clearly is just interesting to ourselves and our little respective coteries of readers. Not to mention, those of us who have been at it for awhile simply have way too much stuff for anyone to actually read.
Also, the danger of reviewing whole blogs is that it comes dangerously close to reviewing whole people. (Which is part of academic life, surely, but let’s leave that kind of stuff to the hiring committees and administration, shall we?) I see your point about how individual posts might have to be seen in the context of a surrounding thought process, but I tend to think that most critically ambitious bloggers occasionally write things with the hope that they might turn out to be of general interest.
Some disjointed notes:
At the journal I review for, we have a rule: no bombs. We do not review books that we are tempted to trash, that induce us to intellectual violence. The journal’s philosophy is: it’s a book, it’s a tool, if you can’t use it, put it down and get another. This rule exists to prevent Established Persons from smacking down Emerging Persons for seeing a point differently or observing methodological commitments anathema to Established Persons. The review space could have a similar rule. Pointing out of flaws is fine, whacking with a cudgel is not.
Reviewers ought not be anonymous. Two reasons: anonymity is a factor permitting trollish behavior; and I, for one, would want to know which reviews were authored by other experts in the area of the post, and which were not. I would also want any admin types to know the difference.
With these two suggestions, I’m heading in a direction that involves some editorial oversight, some one doing that with their time, and that might be something others would want to avoid. How though....?
On reciprocity: Possibly make reviewing mandatory. Post one, review one. No review, then your post and any reviews it received are deleted, no mark against those who did offer reviews. Or, weirdly, the reviews could stay, but your post wouldn’t. Carrot, stick.
I’m also timid of reviewing whole blogs for an additional reason: no writer in any genre hits their best footing every day or in every post. --- Often we really are just noodling around, formulating ideas that we will see we need to drop. Or just venting. Blogs are nice places to discuss that, but do we want promotion resting even in small part on the noodling or venting that we dropped? I doubt it. If I don’t take a post seriously, I don’t want my tenure cmte. to either.
Standardization and risk and play. It strikes me that generically the blog (as has been noted before me) has more in common with the personal essay or the essay in general than with the journal article. Those conventions have been worked out very well in recent years (see Lopate, et alii). These essays easily include intellectual work and subject matter, but they’re written in the process of thinking, not as the conclusions of thinking, the “mess” of the mind at work is part of the pleasure of reading such essays—and blogs. There might be some good guidance there. Plus, some academics have already published, like on paper, a good deal of work in this general genre: Retallack’s *Poethical Wager* or Hejinian’s *The Language of Inquiry* being two examples (of too many, just two I’m working with at the moment).
So, I think that sweet-spot you’re looking for, Amardeep, is completely possible. Might need a little explaining to participants, but completely possible.
What seems particularly workable about this idea is the separation between thought and commerce. That is, an academic can blog about an item of interest or pursue a particular line of inquiry/research without worrying if it is “marketable;” and the larger community (including starving grad students like myself) can read and respond to these ideas without having to purchase expensive books or subscribe to pricey journals.
Not to mention the immediacy of the process—no more time-delay with a publisher.
Blogging is such a completely different model, a new way of making it count is needed. Having voluntary reviewers seems good, but I think it is too much like the system in place for academic journals.
I’ve settled on the notion of real-world relevance. Do people find your interesting and useful enough to read? As long as a blog essay is discipline related (and does not have meta tags for sex, sexy, etc.), then I think page hits or visitor count is adequate. Of course, an alternative would be links to articles. Another site linking to your blog essay is a major vote of respectibility, IMO.
However, it all comes down to popularity. Have our blog posts be read by a couple of thousand (outside of your own locale), and that’s enough.