<< Work in Progress: Humanities Network | Front Page | DeLillo in the Wild >>
Saturday, October 27, 2007
Monkey Grammar
Over at Language Log Geoffrey Pullum has a rant against computerized grammar-checking. He quotes on Jason Snell as saying that “an infinite number of monkeys will analyze your writing and present you with useless grammar complaints.” Pullum concludes:
For the most part, accepting the advice of a computer grammar checker on your prose will make it much worse, sometimes hilariously incoherent. If you want an amusing way to while away a rainy afternoon, take a piece of literary prose you consider sublimely masterful and run the Microsoft Word™ grammar checker on it, accepting all the suggested changes.
Comments
Hmmm, like many, I assume, I run Word’s Grammar Check for a couple of reasons: first, it does find typos, misspelled words, and the occasional unintended grammatical “error”; second, I get to feel superior by clicking “Ignore” repeatedly.
I like that grammar checks are imperfect. If or when they approach perfection, I’ll be seriously bummed, just as I was when Deep Blue beat Garry Kasparov.
Belittling the current state of a technology is always to miss the point. The nature of technology is that version n+1 will probably be a little better than version n.
I’d be interested in a post that argued why computer grammar checking can’t ever be useful. Until then, I’m afraid Geoffrey Pullum’s rant does not rise above technophobia.
(Example: I just corrected a spelling mistake caught by Firefox’s built-in spell checker for text boxes. It’s currently a pretty crappy spell checker, but in a year I bet I’ll be relying on it.)
Don’t confusing spelling checkers with grammar checkers. Finding typos is a good thing.
But English grammar is not something that can really be analyzed programmatically. Believe me, as a professional technology writer (which is why I brought up the subject in the first place) I’m about as far away from technophobia as you can imagine. But as a writer and a technology guy, I can tell you that grammar checkers are a failure because the rules of our language are far too complicated for today’s computer programs to grasp. I’ve yet to see any algorithm that could actually understand sentence and paragraph construction.
Instead, what computer grammar checkers do is look for basic (and debatable if not outright wrong!) “errors” such as split infinitives, run-on sentences (which often aren’t), and the like.
Believe me, on the day that someone comes up with a grammar checker that actually works like an editor or schoolmarm looking over your shoulder, I’ll be the first to turn it on. But right now all they can provide is a false sense of security and constant nudges about things that are probably not wrong. Hence my reference to an infinite number of monkeys. They bring a lot of calculation power to bear, but you end up with more crap on the walls than coherence on the page.
Well you have a point. This seems a matter for a bet that in five years grammar checkers will be useful (or not), but I have no idea what the criteria would be.
But English grammar is not something that can really be analyzed programmatically.
... And even if it were, many of the habits considered “grammatical errors” are better described as “stylistic infelicities” or “correct but clumsy writing”.
Leaving aside the whole question of whether there is an objective grammatical standard to adhere to, most of the rules that find their way into prescriptive grammar should be broken on occasion, because the alternative is worse. (Oops, passive voice… “One should break most of the rules on occasion”?)
The problem with these things is that they’re too ambitious. It’s good to be be told about repeated function words. That’s not spell-checking, but it is catching objective grammatical mistakes, and ones that are often made and surprisingly difficult to spot. But once you start making “rules for grammar checking” you’re more or less committed to trying to address the “common grammatical errors” that people think are there, even if they’re “debatable if not outright wrong”.
(Tom: they’d be useful now if you could choose which individual rules to turn off and which to leave on. Only they’d be doing such a limited task it’s arguable whether we should call them “grammar checkers” or not.)
Last I saw Word’s gramerchek systematically advised at least one error. With subjects like “The men of the family....” Word would sometimes (always?) suggest that the verb agree agree with the word “family”: “The men of the family was gathered in front of the house.”
Iglesias nedes a beter spelchek.
I proofread quite a bit of English prose written by Polish friends. I had a go writing a “depolariser” to add and remove articles as necessary. Unsurprisingly it didn’t work. The results have a weird charm of their own though.
(I was lazy and went for bi- and trigram frequencies; it turns out the dependencies you’re looking for are almost always long-distance so that doesn’t work. But according to Ray Kurzweil you can use them for poetry.)
This study—papyr.com/hypertextbooks/grammar/gramchek.htm— confirms my experience: grammar checkers are sometimes useful.
Obviously, they are tools best used by those who understand grammar.
The study doesn’t count false positives; typically you test algorithms like this on both recall (how many targets did it find?) and precision (how many results returned were really targets?). (That’s the standard for search algorithms, anyway, and seems highly applicable to this case too.)
You can easily improve recall by returning more results, but precision tends to suffer—and while you can easily test recall on a constructed example like this one (with one chosen example of each “error") you really need to test precision on live text.
I don’t know what we are arguing here. Of course grammar checkers are imperfect. If that means one should never use them, then I disagree. Luckily, we can all make our own choices.
Tikitu: I had to teach the use of articles ("a", “an”, “the") to Chinese once, who have nothing comparable. I failed. Everyone fails. Screw the articles.
Every native speaker knows how to use (the?) articles correctly, but no one can explain what they do, prescriptive grammars say nothing because it isn’t a native-speaker mistake, descriptive and analytic grammars say little because it isn’t a deep technical question, and ESL texts seem to neglect the topic.
The dog came into the room.
A dog came into the room.
The dog is a loyal creature.
Dogs are loyal creatures.
NOT: The dogs are loyal creatures.
PROBABLY NOT: The dog came into the room. It was a loyal creature. (Sounds parodic, actually).
Gerrold: you’re right, of course there’s no basic disagreement. And in fact I don’t use a grammar checker because I’m not writing in a word-processor and so there isn’t one there to use… There’s something like an objective question of whether they do more harm than good, especially for people already weak on grammar (that’s where the recall/precision question comes up: if you’re just accepting all changes, do you end up worse than you started?) but I wouldn’t care to take a stand there. And, as you say, probably those people shouldn’t be relying on the tool anyway.
John: Polish (and all the Slavic languages?) has same problem with the articles: they aren’t there.
I was curious, so I went back and looked at how my depolariser was working. Typical example is “finite models corresponding to ***A*** given quantifiers”, which is much more clear-cut. (It would presumably have worked if I’d taken 3/4-grams, since plural “quantifiers” would block the addition. But the sparse data problems, the laziness, etc...) For something it got right (also fairly clear-cut I think, although I can’t tell you what the rule might be): “In ***THE*** second part of the paper examples of syntax learning
models are shown”.
(Starred words are the additions from the algorithm; it seems I didn’t get it to mark deletions at all, which is a shame.)
ALSO NOT: The dog, a loyal creature, came into the room.