Wednesday, December 03, 2008

Reviews symposium: Partly, badly realized

Let's all welcome Shawn Elliott back to Blogistan. Hopefully they're not working his fingers to the bone at 2K Boston, and he'll have plenty of time to favor us with some posts. In the inaugural post at his new blog, Shawn posits a series of questions about the ethics and practices of game reviews. It's interesting stuff, and fodder for serious discussion. (Just like a LiveJournal meme!) I figured I'd go ahead and answer his questions. Maybe some of you will have reactions of your own to share in comments.
Question 1: How much is on our minds before we begin playing any given game for review purposes? Will we imagine a range of probable scores that a heavily marketed, highly budgeted, and hugely anticipated game will get? What when the game is branded "budget" or is the work of a lesser-known, less-storied studio? If so, how closely have actual scores correlated with our assumptions?

As little as possible, if I can help it. Generally the games we opt to cover at the Phoenix are high-profile ones, although it's nice once in a while to seek out something a bit more obscure. Even so, I try to avoid reading too much preview copy, or knowing more than the bare minimum about a game before I play it. That seems like the best way to be fair. Sometimes a little foreknowledge is unavoidable, though, such as when playing a sequel or a game from a distinctive development house. I think that's okay, particularly if you mention it during the review.

Question 2: Ought reviewers settle on a score before, during, or after writing a review? How consistent are our practices with our prescriptions? Have we, for instance, revised a score after writing our reviews, even though we advocate against it, and if so, why?

I don't know how much it matters. Most of the time I assign the score before I write the text, although there have been numerous occasions in which I've changed the score after writing the review. Usually it's a matter of a half a point, but it's happened on both ends of the scale. I've written reviews and thought, "This sounds like a 9.0 instead of an 8.5," and I've said, "This sounds like a 4.0 instead of a 4.5." Sometimes the process of writing a review can clarify your thoughts about a game in a way that thinking about it cannot. Reviewers ought to do what works for them. (I don't see why you'd advocate against changing your score, either.) If the review and the score don't seem to agree, one of them should be changed.

Question 3: When possible, do we look at the scores that other critics give to the games that we're reviewing, as we review them? If so, are groupthink or iconoclasty potential problems?

Never. Never. I go out of my way to avoid reading other reviews or looking at a game's Metacritic score until after I've written my own review. Whether you think you're impervious to this kind of influence, it's got to stick in your mind at least a little bit if you disagree with the masses. Why take the risk? 9 times out of 10 it seems like my score is right on the Metacritic money, but that 1 other time is where your integrity lies.

Question 4: Often times we will have repeatedly played and/or previewed games in development prior to reviewing them. Does this familiarity with a particular game's developmental process influence the scores that we assign to the final product in the way that a professor will take into consideration her students' limitations and proven potential when she evaluates papers at the end of the semester?

This has never come up. In fact, I'm still angry at Harmonix for not showing me the Guitar Hero prototype that must have been somewhere in their office when I visited. Frankly, I'm not sure how I'd handle such a situation. On one hand, a better knowledge of the challenges developers face might help me write more informed reviews. On the other hand, it's not my job to review what should have been, or what I know the developer wishes had been. I'd like my first exposure to a game to be its completed form.

Question 5: Review writing carries real consequence, especially among members of the enthusiast press. Once-warm PR people and game producers can become cold upon our publication of undesirable review scores, diminishing or eliminating our ability to secure subsequent interviews and access. Postmortem discussions and exclusive looks at the publisher and/or developer's forthcoming products are less likely. Conversely, a few publishers will permit us to post reviews before competitors, provided our review scores are favorable. Do such pressures produce a subliminal background or even enter our thoughts as we write reviews and assign scores?

Again, hasn't come up -- one of the benefits of writing for non-enthusiast publications. I have to say, I don't know how I'd handle it if I had actual relationships with PR folks. I have a horrible tendency to tell people what I think they want to hear.

Question 6: Is grade inflation an ongoing problem?

Less so than the habit of declaring games "one of the best," "one of the most," or "important," before we've even had a chance to let the disc drive stop spinning. Look, scores probably are too high. Everyone seems to agree on this. But I think the Metacritic model has largely done away with this problem. Can you tell the difference between a score of 9.6 and 9.7? Not on one site, no. But in the aggregate, yes. That is, a 10-point difference in a Metacritic score usually seems definitive to me, even if both games are scored too high.

Question 7: Do scores determine our tone? Can a "3" encourage us to explain an aspect of a game in clearly negative terms where our attitude is actually less decided? Example: Game X's camera obscures the action, combat is irritatingly difficult, and "save" stations are few and far between. In our reviews, is Game X's plot, which we're still thinking through, more likely to become miserable than plain?

In this case, I would probably skip discussion of the plot altogether, or mention it briefly as one case in which the game is not a total loss. Here's why: In this scenario, the game's plot, however strong, obviously could not overcome the game's other flaws. Therefore, who cares about it? It may be worth the mention for readers who might be interested, but, personally, I don't think it's the critic's job to try to divine what hypothetical human being might enjoy each and every game. It's the critic's job to be honest about his experience.

Question 8: Do scores encourage our readers to conduct a sort of text-to-number calculus where the two obviously negative statements in an otherwise positive-sounding review necessarily translate into every point deducted from the "10" that the game didn't get? Does this make reviews with high marks more likely to overlook fault, and reviews with low marks less likely to celebrate accomplishment?

If so, whose problem is that? But just as with the situation above, if a reviewer thinks a game deserves a 9 or a 10 despite some fault or another, I'm not sure how important it is to mention that. This does get at the basic problem of the scoring system in general. It's important to remember how much people really like scores, though, and how useful aggregators have turned out to be. I don't think the solution is to throw out scores entirely, tempting though it may sometimes be. The best a reviewer can do is try to be consistent. (Note: No one will ever accomplish this.)

Question 9: Which is more important to us, our scores or our copy? If the latter, have our responses revealed any inconsistencies between our attitudes and actions? Are we still convinced of the importance and power of scores?

Copy, copy, copy. Any writer would tell you that. Nothing is more important to me than the text of my own reviews. But I'll let the readers tell me whether my responses have revealed inconsistencies between my attitude and my actions. I don't think so, but I'm not the best judge of that.

Related suggestions for Ethics section:
Have we ever submitted review scores to publishers prior to their publication? If so, why?


Have we ever submitted review copy to publishers prior to its publication. If so, why?


Have PR people suggested that specific critics review specific games? Have we complied with their suggestions?


Reviews Vs Criticism
Question 1: What is the object of a review? What are the review writer's obligations?

This depends on the outlet. I actually do think the big sites should be acting as consumer advocates, even if I wish they were a little bit more adept at doing so. For myself, and for the kind of writing I find most interesting, I think the point of a review is to illuminate what a game is actually about, beyond the surface level of the plot and the play mechanics. How do they intersect? How do the game's apparent goals seem to mesh or conflict with its execution? What is this game trying to say? A reviewer's obligation is an honest and thorough attempt to find both the question and the answer.

Question 2: If the purpose of a review is to suggest to consumers how they should spend their time and money, why do we avoid less-granular grading scales such as Buy, Try, or Avoid? Example: Giant Bomb founder and former Gamespot editorial director Jeff Gerstmann told MTV's Multiplayer blog that “'How can I save people money today?' is basically the kind of mentality that I tackle this stuff with.” Under Gerstmann's directorship, Gamespot reviewed games on a hundred-point scale. Is a 9.6 different than a 9.7 when the wisdom of a purchase is what the reviewer wants to communicate?

Well, I don't think this is the purpose of all reviews. Like I said, for Gamespot or IGN, it probably is, and that's valuable in its own right. But for what I'm after in reviews, it doesn't matter if a game costs $100, or is free. I want what I say about it today to be true whether it's brand-new, part of a "Greatest Hits" collection, or even eventually available for free as abandonware. I probably don't always meet that goal, but it is the goal.

Question 3: Actual sales rarely correlate with review scores in cases where games are not also heavily hyped and marketed. Increasingly, gamers pre-order games prior to the publication of reviews. Interactive demos allow our audiences to decide for themselves whether or not a game will be worth their dollars. In addition, word of mouth and message board discussions inform our potential audiences' purchasing decisions with an intimacy and directness that we cannot provide. Finally, review aggregation sites such as Metacritic mute the bias of individual reviewers and provide a bigger picture. Do these circumstances suggest that our self-perception is, well, delusional – a throwback to a time when magazines and websites were gaming's gatekeepers? If our audiences believe this, even if we do not, what are they really reading for?

I don't think my self-perception is what Shawn implies here. I've had too many conversations with Phoenix readers who weren't aware that the paper even runs game reviews to get too full of myself. But this is part of the argument for deeper, better game reviews. It's good that gamers can decide on their own what to get, thanks to demos. It's not so good that a well-oiled PR campaign can turn an iffy game into a million-seller. If reviewers want to consider themselves gatekeepers, they are doing a very bad job of it -- numerous and varied counterexamples not withstanding.

Question 4: Can criticism (concerned with telling our audiences what they're spending time and/or money playing as opposed to whether or not a game is worth spending time and/or money to play) coexist with reviews? Is a competent review also a critique -- as is so often the case where lit, movies, and music are concerned -- or should we separate the two?

I think the best reviews would have a little from column A and a little from column B. If a review is written well enough, even if it leaves out some things that might be important, a reader should still get an idea of the game's gestalt experience. They may read a positive review and be convinced that a game is not for them, or read a negative review that nevertheless convinces them to give the game a chance. The goal isn't for readers to agree with everything the reviewer says, but for the reviewer to provide food for thought.

Question 5: What can (or should) such criticism take into account?

I think I got to this up above somewhere, but to distill it, I'll paraphrase Roger Ebert: "What is this game about, and how is it about it?" I'm still irked by criticisms of No More Heroes that complained about jagged graphics and the limits of the Santa Destroy overworld. The game was a satire. Those things were manifestly the point of the game. Now, a reviewer may still fairly dislike the game for these reasons and more, but it was a dereliction of duty not to accurately interpret the game's intent. On a broader scale, I believe games have a responsibility to be user-friendly, thoughtful, and moral (not moralistic). These are all things I try to think about when I write a game review.


Etelmik said...

You know, I think I'm going to do this too.

Mitch Krpata said...

Tag, you're it!

Julian said...

For what it's worth, as a consumer and not producer of reviews I pay close attention to the copy of the review, even when I'm attempting to make a buy/don't buy decision. I have somewhat atypical tastes and very strong pet peeves, in addition to tending to be a little lenient on execution if the game is inventive and fun. So I know what I'm looking for in the body of the text and I've been completely sold on a game with a 7 (you know, the low end of the scale...), and completely turned off by high-scoring reviews. I generally prefer review styles that lay out positives and negatives clearly (or high points and low points like AV Club's reviews), and use condensed review scales (letter grades or 1-5 are fine).

Etelmik said...

Julian, you must like Mitch's reviews then!

And Mitch, sorry if it looks like I'm going to steal the thunder; it's just that I've written and edited a lot of reviews in the last two months (12 and 53) and then I interviewed you both and N'gai and...yeah.

Mitch Krpata said...

I hope a lot of people do! I'd like to see the different approaches people take. It's possible some other people have already done it, even.

Anonymous said...

On that last point, do you feel that the reviews that mentioned poor gunplay in Mirror's Edge were a dereliction of duty? After all, the game was trying to be Run Lola Run, not Hardboiled. Running was the point, shooting wasn't.

When we do a podcast over at the Squadron of Shame, we like to ask "What was the Thesis of this game? What were the developers trying to do, and were they successful?". Do you think that's a good question that every reviewer should ask themselves?

Mitch Krpata said...

I don't think it's bad to mention it, no, since it is fairly weak, but certainly it's not a great way to criticize the game. It seemed clear to me that trying to play without even using the guns was the better way to go. More important to mention the bad fighting, since there's one part where that's mandatory (and so annoying!). But sure, if someone wrote a negative review saying, in essence, "this was a terrible first-person shooter," you might think they weren't doing their job very well.

That sounds very much like a question people should ask, although personally I'm not so concerned with authorial intent. Maybe the developers were trying to do one thing and succeeded in doing something else entirely. I'm more concerned with what it is their game seems to be trying to do. (Is that too fine a distinction?)

Etelmik said...


I hope anyone who writes in games publications writes something on this and then readers see it and think "You know I'd like more criticism" and then BOOM YEAH AWESOMEEEE etc.

Anonymous said...

In your response to question 5, you said:

"I'm still irked by criticisms of No More Heroes that complained about jagged graphics and the limits of the Santa Destroy overworld. The game was a satire. Those things were manifestly the point of the game. Now, a reviewer may still fairly dislike the game for these reasons and more, but it was a dereliction of duty not to accurately interpret the game's intent."

I was just curious as to what you think of Alex Kierkegaard's (from rebuttal.

Mitch Krpata said...

I think it's a lot better than the IGN review I was referring to! Plus, Kierkegaard clearly gets it -- he just doesn't like it. Fine by me.

But I disagree with him in that I thought the game was a blast to play, particularly the fighting. I even enjoyed the part-time job minigames, which were parodies both of theme and mechanics.

(Here was my review of No More Heroes, btw.)

Iroquois Pliskin said...

I thought this was a great idea for a post. A lot of questions, it seems to me, come from this enthusiast-press anxiety about reviews being a cog in the promotional machine, it seems like you get more chance to be a critic writing for a mainstream publication like the Phoenix.

I like the ebert quote in the last paragraph as a critical credo, I think approach is a good way to bridge that consumer advocate/critic dichotomy that people worry about. Usually what makes a game fun (purchase-worthy) is its success in achieving what it sets out to do.

Anonymous said...

I agree with the general assertion that game review scores are ridiculously high on average, but I think a part of that is the result of games being a medium in which the actual product shipped can be irreparably broken (one of the most famous examples being that Big Rigs racing game where the opponents never leave the starting line). No other medium would put up with that; how would you review a movie that froze up and refused to play the last third, or had a glitch where the antagonist didn't appear in key scenes despite their presence being very clearly required? Because of the potential for this in the games industry, there absolutely must be some method of heaping the appropriate scorn on a fundamentally broken product, and the best method games journalism has come up with is reserving the bottom end of the review scale. The only other real option is to completely ignore those games (much as film reviewers skip direct to video movies), but then because of the fact that the product itself maybe broken that serves as an abrogation of the consumer advocate roles of the games press as well. In order to replace the way the scales are used, some method of retaining the exposure of broken products must be formulated.