So, after an unexpected vacation, work continues. Sort of. A common request was for the ability to compare an outlet internally against itself to see how it reviews games on different platforms. I’m still working on a dynamic means to do this, and one which will fit a smooth curve to the data points. But in the meantime I took the Gamespot data and did the necessary database query for PS2, ‘Cube, Xbox, and PC and slapped them together manually into this animated GIF.

Is Gamespot “BIAS”, as the kids who haven’t learned the adjectivial form of the root word like to say? That you’ll have to decide for yourself. I can tell you the 23 scores in there that have my name attached to them are individually the one true score for each game they were given to.

But check out the weird distribution on Gamecube scores. The most common score is 6.5, with 19 games scoring 6.5. But, the next most common review score is a tie between 8.0 and 7.2, with 18 games scoring thusly. There’s no clear tip of the curve, but one thing to keep in mind is that there are a lot fewer datapoints for Gamecube, with only 442 reviews compared. The average score per platform comes out to be almost the same— in between 6.9 and 7.0— except for PC, where it’s a lot more profitable to publish crappy budgetware.

Another weird thing you might notice is how certain scores tend to not be represented in the expected curve. In particular notice how no matter which console platform you look at, it seems particularly difficult to score a 7.7 or a 7.8. There’s a noticeable dip at 7.4 too. PC seems more immune to that phenomenon. Like I said, weird.

Anyway, have fun comparing and contrasting each and every point on the scale, to prove whatever point you think needs making.

August 21st, 2006 by MJG
Tagged: Editorial
          

Hi there. The stuff you’ve been linked to probably involves the next two articles after this one. Please also enjoy these fine articles linked below, even though they have less interesting pictures and involve far fewer, if any, numbers.

—Matthew J. Gallant

• Your Guide To The Cherokee People (Look for “Your Guide To Ancient History” in the next issue of Computer Games Magazine)
• Videogames Are Art
• Alyx Is A Nerd Succubus
• Rockstar Cancels A Game
• Documentaries On Making Games: We Care
• Half-Life: 2, Episode: 1, Boats & Buggies: 0
• $600 PS3 Is Not A Toy; Therefore, $650 PSP Watch Is Not A Timepiece.

August 11th, 2006 by MJG
Tagged: Editorial
          Comments Off

Well, it was significantly harder to grab IGN’s scores because of their tendency to throw interstitial ads at you. But it’s done. The histogram reveals IGN’s weirdo 20-point/100-point scale waffling. I’m tempted to round their scores to the nearest .5 and force them onto a 20-point scale.

Still, their most common score being an 8.0 is revealing. Not to mention there being almost as many 9.0’s as there are 6.0’s.

Now that I have two review outlets in the database, I’m going to spend more time working on the presentation of the site and the tools that will let users get more detailed information for one single site (like comparing score breakdowns by platform), as well as comparing between sites.

Related - Kotaku Doesn’t Read The Articles They Link

August 6th, 2006 by MJG
Tagged: Editorial
          1

I spent the last day or so constructing the database and writing a script that would grab all of Gamespot’s reviews from their review list and massage the info into the relevant database tables.

Then I wrote the script that generates the image you see above. It’s a histogram of all Gamespot’s review scores for any game that isn’t for mobile phones, the Gameboy/Gameboy Color, the N-Gage, or the Neo-Geo Pocket. I’ll be expanding its functionality to make comparisons by platform as well as between two or more different sites. And I suppose I’ll have to make it look nicer, too.

I did this really quickly because I wanted to see if this was really worth doing. Out of the 7,244 reviews, 239 scored a 7.0. There’s some weird gaps, but you can definitely see the bell shape and how scores crowd towards the right of the scale. Now to figure out a way to determine which ranges of scores get translated to 1, 2, 3, 4, and 5 stars. It looks like that a 3-star game may go as high as 8.1 or 8.2.

Hopefully review scores from other sites will be as easy to compile as Gamespot’s were. I don’t know if any other sites have a list of all their reviews like that.

August 5th, 2006 by MJG
Tagged: Editorial
          Comments Off

The site will be undergoing a radical transformation. It will no longer be a blog. There will be a different sense of humor to the site. I think it will be even more funny, just because of what it will do to both the game reviewing and game publishing industries.

The joke is this: lots of people love sites like GameRankings and MetaCritic. Publishers and fanboys alike use a game’s average score like a weapon. But really, these sites tend to not provide useful information about a game’s review scores. Why? For a couple of reasons.

Blame falls mostly on the reviewers themselves. Most reviewers tend to artificially inflate a game’s score. This has become known as the 7-9 scale, so named due to the industry’s stubbornness in holding onto a 100-point scale, and it’s further stubbornness in using any parts of it that don’t fall between 72 and 95. A famous “IGN 9.2″ tends to have a lot of noise in its signal.

Further fault, however, lies with GameRankings and its ilk themselves, though. They tend to be more than a little forgiving when it comes to who’s allowed to contribute to the “averaged score”. The criteria for GameRankings comes down to a site needing to have a lot of reviews and having a current output of a certain number of reviews a month. Very egalitarian, but there are way too many sites run by, shall we say, the overeager. There are 34 sites with an average review score of 80 or higher. I’ll rephrase that because it bears repeating: GameRankings has 34 sites— that it uses to generate a game’s metascore— for whom the average game scores an 80. Or higher— the creampuff champ is PSE Magazine, which has 817 reviews, the average score of which is 86.7.

Metafuture will improve the service that sites like GameRankings provide. Put a better way, we will create the service that those sites attempt to provide. First of all, we’ll cut down on the number of review outlets. There’s no need to have a huge number of reviews for a game to provide a useful aggregated score. Magazines and the top websites will be just fine.

Second comes the funny part. Games will still get an average score from all contributing reviews. But a site’s contribution to that average will depend on that site’s own individual normal curve— with the immediate left and right of the bell’s tip signifying three stars on a scale of one to five. Watch the drama as the biggest sweethearts see their 8.4 score for Gun and Car IV get pegged as three stars.

Continue to subscribe to this blog for status updates; the blog will only truly die when the new site is ready to launch, and compiling data is going to take a significant amount of time. If you’re an editor of a review outlet or otherwise have access to raw review data, please contact the site. I won’t blame you if you don’t, though.

August 4th, 2006 by MJG
Tagged: Editorial
          Comments Off

Mark Rein engages in a bit of punditry in this interview at Eurogamer.

Developers did not have finished Xbox 360 hardware last year at E3. So Sony’s actually maybe in a better place vis-a-vis Microsoft in relation to launch.

Yeah, Mark. Except Sony’s launch window was three months ago.

Oh right, I’m talking to Mark Rein.

Yeah, Mark. Except Sony’s launch window was… at least two weeks from the end of April.

July 26th, 2006 by MJG
Tagged: Sony, XBox 360, PS3, Mark Rein
          Comments Off

In our rundown of Japan’s blossoming love affair with Medieval and Revolutionary France, Metafuture regrets being remiss in not including what may be the seed: Arm Joe, the 2D fighter inspired by one Japanese man’s love of Les Misérables.

This man, known only as Takase, has been working on this PC game all by himself since the late 90s. We saw this get mentioned as recently as last month on GameSetWatch, but it completely slipped our minds.

To make up for it, here’s a link to download the game, if you haven’t yet seen Robo-Valjean avenge the mistreatment of his human peasant buddies. (The download will start automatically)

Metafuture infobit: It’s called Arm Joe because in Japan, Les Misérables is called Ah, Mujou (”Ah, Cruelty”).

July 24th, 2006 by MJG
Tagged: PC
          Comments Off

First there was that X360 game about our old half-French buddy Chopin.

Then last week I saw an anime about what would happen if Louis XV, instead of saying “après moi, le déluge,” instead said “pendant moi, le GODDAMN ZOMBIES VS. GENDER-CHANGING PALADINS”. Actually, it was just one dude being possessed by his dead sister. And it was really cool up until that point.

Now Level 5— the developers of Dark Cloud, Dragon Quest VIII, and Rogue Galaxy— take us way back and turn Joan of Arc into an SRPG for the PSP. If there is no North American release planned, may Level 5 put one there, and if there is, may Level 5 so keep it.

July 24th, 2006 by MJG
Tagged: PSP, Level 5
          Comments Off