Reviews symposium: Partly, badly realized

Let's all welcome Shawn Elliott back to Blogistan. Hopefully they're not working his fingers to the bone at 2K Boston, and he'll have plenty of time to favor us with some posts. In the inaugural post at his new blog, Shawn posits a series of questions about the ethics and practices of game reviews. It's interesting stuff, and fodder for serious discussion. (Just like a LiveJournal meme!) I figured I'd go ahead and answer his questions. Maybe some of you will have reactions of your own to share in comments.

Question 1: How much is on our minds before we begin playing any given game for review purposes? Will we imagine a range of probable scores that a heavily marketed, highly budgeted, and hugely anticipated game will get? What when the game is branded "budget" or is the work of a lesser-known, less-storied studio? If so, how closely have actual scores correlated with our assumptions?

As little as possible, if I can help it. Generally the games we opt to cover at the Phoenix are high-profile ones, although it's nice once in a while to seek out something a bit more obscure. Even so, I try to avoid reading too much preview copy, or knowing more than the bare minimum about a game before I play it. That seems like the best way to be fair. Sometimes a little foreknowledge is unavoidable, though, such as when playing a sequel or a game from a distinctive development house. I think that's okay, particularly if you mention it during the review.

Question 2: Ought reviewers settle on a score before, during, or after writing a review? How consistent are our practices with our prescriptions? Have we, for instance, revised a score after writing our reviews, even though we advocate against it, and if so, why?

I don't know how much it matters. Most of the time I assign the score before I write the text, although there have been numerous occasions in which I've changed the score after writing the review. Usually it's a matter of a half a point, but it's happened on both ends of the scale. I've written reviews and thought, "This sounds like a 9.0 instead of an 8.5," and I've said, "This sounds like a 4.0 instead of a 4.5." Sometimes the process of writing a review can clarify your thoughts about a game in a way that thinking about it cannot. Reviewers ought to do what works for them. (I don't see why you'd advocate against changing your score, either.) If the review and the score don't seem to agree, one of them should be changed.

Question 3: When possible, do we look at the scores that other critics give to the games that we're reviewing, as we review them? If so, are groupthink or iconoclasty potential problems?

Never. Never. I go out of my way to avoid reading other reviews or looking at a game's Metacritic score until after I've written my own review. Whether you think you're impervious to this kind of influence, it's got to stick in your mind at least a little bit if you disagree with the masses. Why take the risk? 9 times out of 10 it seems like my score is right on the Metacritic money, but that 1 other time is where your integrity lies.

Question 4: Often times we will have repeatedly played and/or previewed games in development prior to reviewing them. Does this familiarity with a particular game's developmental process influence the scores that we assign to the final product in the way that a professor will take into consideration her students' limitations and proven potential when she evaluates papers at the end of the semester?

This has never come up. In fact, I'm still angry at Harmonix for not showing me the Guitar Hero prototype that must have been somewhere in their office when I visited. Frankly, I'm not sure how I'd handle such a situation. On one hand, a better knowledge of the challenges developers face might help me write more informed reviews. On the other hand, it's not my job to review what should have been, or what I know the developer wishes had been. I'd like my first exposure to a game to be its completed form.

Question 5: Review writing carries real consequence, especially among members of the enthusiast press. Once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background or even enter our thoughts as we write reviews and assign scores?

Again, hasn't come up -- one of the benefits of writing for non-enthusiast publications. I have to say, I don't know how I'd handle it if I had actual relationships with PR folks. I have a horrible tendency to tell people what I think they want to hear.

Question 6: Is grade inflation an ongoing problem?

Less so than the habit of declaring games "one of the best," "one of the most," or "important," before we've even had a chance to let the disc drive stop spinning. Look, scores probably are too high. Everyone seems to agree on this. But I think the Metacritic model has largely done away with this problem. Can you tell the difference between a score of 9.6 and 9.7? Not on one site, no. But in the aggregate, yes. That is, a 10-point difference in a Metacritic score usually seems definitive to me, even if both games are scored too high.

Question 7: Do scores determine our tone? Can a "3" encourage us to explain an aspect of a game in clearly negative terms where our attitude is actually less decided? Example: Game X's camera obscures the action, combat is irritatingly difficult, and "save" stations are few and far between. In our reviews, is Game X's plot, which we're still thinking through, more likely to become miserable than plain?

In this case, I would probably skip discussion of the plot altogether, or mention it briefly as one case in which the game is not a total loss. Here's why: In this scenario, the game's plot, however strong, obviously could not overcome the game's other flaws. Therefore, who cares about it? It may be worth the mention for readers who might be interested, but, personally, I don't think it's the critic's job to try to divine what hypothetical human being might enjoy each and every game. It's the critic's job to be honest about his experience.

Question 8: Do scores encourage our readers to conduct a sort of text-to-number calculus where the two obviously negative statements in an otherwise positive-sounding review necessarily translate into every point deducted from the "10" that the game didn't get? Does this make reviews with high marks more likely to overlook fault, and reviews with low marks less likely to celebrate accomplishment?

If so, whose problem is that? But just as with the situation above, if a reviewer thinks a game deserves a 9 or a 10 despite some fault or another, I'm not sure how important it is to mention that. This does get at the basic problem of the scoring system in general. It's important to remember how much people really like scores, though, and how useful aggregators have turned out to be. I don't think the solution is to throw out scores entirely, tempting though it may sometimes be. The best a reviewer can do is try to be consistent. (Note: No one will ever accomplish this.)

Question 9: Which is more important to us, our scores or our copy? If the latter, have our responses revealed any inconsistencies between our attitudes and actions? Are we still convinced of the importance and power of scores?

Copy, copy, copy. Any writer would tell you that. Nothing is more important to me than the text of my own reviews. But I'll let the readers tell me whether my responses have revealed inconsistencies between my attitude and my actions. I don't think so, but I'm not the best judge of that.

Related suggestions for Ethics section:
Have we ever submitted review scores to publishers prior to their publication? If so, why?

No.

Have we ever submitted review copy to publishers prior to its publication. If so, why?

No.

Have PR people suggested that specific critics review specific games? Have we complied with their suggestions?

No.

Reviews Vs Criticism
Question 1: What is the object of a review? What are the review writer's obligations?

This depends on the outlet. I actually do think the big sites should be acting as consumer advocates, even if I wish they were a little bit more adept at doing so. For myself, and for the kind of writing I find most interesting, I think the point of a review is to illuminate what a game is actually about, beyond the surface level of the plot and the play mechanics. How do they intersect? How do the game's apparent goals seem to mesh or conflict with its execution? What is this game trying to say? A reviewer's obligation is an honest and thorough attempt to find both the question and the answer.

Question 2: If the purpose of a review is to suggest to consumers how they should spend their time and money, why do we avoid less-granular grading scales such as Buy, Try, or Avoid? Example: Giant Bomb founder and former Gamespot editorial director Jeff Gerstmann told MTV's Multiplayer blog that “'How can I save people money today?' is basically the kind of mentality that I tackle this stuff with.” Under Gerstmann's directorship, Gamespot reviewed games on a hundred-point scale. Is a 9.6 different than a 9.7 when the wisdom of a purchase is what the reviewer wants to communicate?

Well, I don't think this is the purpose of all reviews. Like I said, for Gamespot or IGN, it probably is, and that's valuable in its own right. But for what I'm after in reviews, it doesn't matter if a game costs $100, or is free. I want what I say about it today to be true whether it's brand-new, part of a "Greatest Hits" collection, or even eventually available for free as abandonware. I probably don't always meet that goal, but it is the goal.

Question 3: Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture. Do these circumstances suggest that our self-perception is, well, delusional – a throwback to a time when magazines and websites were gaming's gatekeepers? If our audiences believe this, even if we do not, what are they really reading for?

I don't think my self-perception is what Shawn implies here. I've had too many conversations with Phoenix readers who weren't aware that the paper even runs game reviews to get too full of myself. But this is part of the argument for deeper, better game reviews. It's good that gamers can decide on their own what to get, thanks to demos. It's not so good that a well-oiled PR campaign can turn an iffy game into a million-seller. If reviewers want to consider themselves gatekeepers, they are doing a very bad job of it -- numerous and varied counterexamples not withstanding.

Question 4: Can criticism (concerned with telling our audiences what they're spending time and/or money playing as opposed to whether or not a game is worth spending time and/or money to play) coexist with reviews? Is a competent review also a critique -- as is so often the case where lit, movies, and music are concerned -- or should we separate the two?

I think the best reviews would have a little from column A and a little from column B. If a review is written well enough, even if it leaves out some things that might be important, a reader should still get an idea of the game's gestalt experience. They may read a positive review and be convinced that a game is not for them, or read a negative review that nevertheless convinces them to give the game a chance. The goal isn't for readers to agree with everything the reviewer says, but for the reviewer to provide food for thought.

Question 5: What can (or should) such criticism take into account?

I think I got to this up above somewhere, but to distill it, I'll paraphrase Roger Ebert: "What is this game about, and how is it about it?" I'm still irked by criticisms of No More Heroes that complained about jagged graphics and the limits of the Santa Destroy overworld. The game was a satire. Those things were manifestly the point of the game. Now, a reviewer may still fairly dislike the game for these reasons and more, but it was a dereliction of duty not to accurately interpret the game's intent. On a broader scale, I believe games have a responsibility to be user-friendly, thoughtful, and moral (not moralistic). These are all things I try to think about when I write a game review.