Microsoft Silent as Xbox Indies Marketplace Exploited

There have always been rumors that the game ratings on the Xbox 360 and Xbox.com website we’re being manipulated by game developers.  On March 23, 2011 rumor became fact when Robert Boyd noticed his game Cthulhu Saves the World slipped from #6 on the top rated list to #11 in a week and showed a suspicious jump in the total number of votes.  During the same time frame the game College Lacrosse 2011 rocketed into the Top 10, propelled by a request to fans to rate the game 5-stars on FaceBook.

The conversation has mostly moved around which developers were suspect.  Who was up voting their game, and who was down voting other games.  Cthulhu Saves the World now sits at #29 in the Top Rated list.  To put this into perspective, I asked Robert about the change in daily sales since his games have changed position.

The ratings started dropping in the middle of March so I've provided
our daily averages for February & April.

70 BoDVII average - Feb
40 BoDVII average - April

83 CSTW average - Feb
53 CSTW average - April

The CSTW drop is likely mostly due to it being a new game and
naturally dropping over time. However, the BoDVII drop is much more
obvious since the game had been out for nearly a year before this
whole mess.

BoDVII is Breath of Death VII.  Doing some armchair math, we can put a dollar figure of sales potentially lost by this change.

  Feb Apr Cng Price Day Loss Month Loss
BoDVII 70 40 30 $1.00 $30.00 $900.00
CStW 83 53 30 $3.00 $90.00 $2,700.00
            $3,600.00

$3,600.00 USD is not much to Microsoft, but that is a huge chunk to an independent developer.

Manipulation is Wide Spread

I used Robert’s games as an example of the impact, but it’s not limited to just a few games.  Rampant down voting has been present in all the top titles.  Here are the number of games over time with 4 star or better rating.

Week / Rating 4 4.25 4.5 4.75 Total
2/27/2011 62 17 11 1 91
3/6/2011 64 16 11 1 92
3/13/2011 68 14 11 1 94
3/20/2011 64 13 10 1 88
3/27/2011 62 7 8 1 78
4/3/2011 54 6 6 1 67
4/10/2011 37 11 2 1 51

Before the ratings manipulation we had over 90 games with 4 stars or better, now it’s just 50.  It’s clear looking at the volume of votes there is a good deal of ratings warfare happening, unchecked.  I have data of games by week, the ratings they’ve held, how much they’ve changed, and what the average vote would be to cause that change.  I wrote a bit on this last week.  I’m not posting the list of games (yet) as I don’t want to focus the conversation on developers.  The focus needs to remain on Microsoft, who continue to stand by and do nothing while this takes place.  I will talk about one game however.

Anatomy of an Exploit

I stressed on my numbers post, in the community forums, and directly in email to Microsoft how simple the ratings exploit is.  It’s clear looking at the data that it’s well known.  Some think you need legions of fans, willing to vote at xbox.com, and set aside morals to attack other games.  Others believe you need to create a voting bot to automate the process.  I believed it was much simpler than that, and I set out to test this theory Friday.  I shocked even myself how easy it was.

The steps:  Visit xbox.com and click on “Sign Up”.  On the first screen in the email box, enter a fake email address.  On the second screen fill out the form with fake data and complete the captcha.  Once you click okay, you now have a fake account that can rate games.  There is no verification of Xbox Live Accounts.

I used an form auto-complete extension for FireFox so the only thing I had to do manually was enter the captcha.  I did not use any automation software, not even Fiddler to replay an http request.  I could cast a vote for my game IncaBlocks every 20-30 seconds, depending on how many times I messed up the captcha.  I spent a few minutes in the morning voting, a few before lunch, and a few before I went home.  About an hour spent in total, and I had calculated 100 votes would move us from below #1400 to just around #100.

Like that guy in office space, I must have made a decimal point error somewhere.  Saturday IncaBlocks launched to #28 in the Top Rated list.  One guy, with a web browser and 60 minutes of spare time can move a game from the bottom of the pack to the front. 

(The point here is to show how easy it is to manipulate the lists, but to share the rest of the IncaBlocks story we went from getting 0 to 1 trial downloads in a day to 15 trial downloads in a day, and no sales in either case.  No amount of rating manipulation is going to make IncaBlocks a fun game!  I’m also okay with this, it was a learning experience and that experience is payment enough.  If Microsoft decides to pull IncaBlocks over this, it never made the $150 requirement to get paid anyway and I’m fine using it to force Microsoft into action.  It would prove Microsoft is capable of taking action, even if they haven’t.)

Microsoft Responds, but only to WP7 Issues

Around the same time as this issue was found, a WP7 top lists issue was reported.  Microsoft responded the same day as it was reported with a personal email address and phone number.  “Top Men” are assigned the case and it’s fixed in a day.

The Xbox Indie problem so far has only gotten a two tweets from Microsoft, and a post by a developer that Microsoft is still trying to figure out who should be in charge of the issue.  (I must blog too much as Microsoft doesn’t respond to my emails *sniff*).

Both WP7 and Xbox use the same site “AppHub” for support.  It’s clear Microsoft has budget to support WP7, but none to support XBLIG.  Even if they cannot decide who is in charge, they can shut down the exploit at Xbox.com by temporarily removing the ability to rate.

The system of exploiting ratings though Live accounts is large.  There is a story breaking now that a design firm may have commissioned 5,000 fake accounts for manipulation.

My Personal Thoughts

In all of this I have one big thought – if Microsoft cannot manage a small group of Xbox Indie games, why would I ever get involved in the larger WP7 Marketplace?  How do I know that in a year the WP7 Marketplace won’t be neglected like XBLIG?  In both marketplaces, WP7 and XBLIG, Microsoft holds back features for Microsoft Partners and sends me the message they are not as interested in developers as they are in publishers.  This is just my opinion, and others will disagree, but right now I’m not feeling Microsoft understands the needs of independent developers and are only interested in giving their big partners a heavy advantage.


Posted 04-11-2011 3:14 PM by Michael C. Neel

[Advertisement]

Comments

Wonder wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 7:06 AM

As always Micro$oft way rule ! lulz.

david wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 7:27 AM

damn right about "if Microsoft cannot manage a small group of Xbox Indie games, why would I ever get involved in the larger WP7 Marketplace..."

[ gotta admit, i was starting to wonder "so what..." until i read that (probably cos i'm not so smart).  but the dude's certainly got a point ]

ThatOtherGuy wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 9:09 AM

Prime case of Microsoft's fickleness. They need to decide what stuff they're going to drop and what stuff they're going to support without making their freaking customers play this guessing game. I could go into a programmer rant about LINQ to SQL, Silverlight, among others, but I digress.

Daniel wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 10:38 AM

Now that Microsoft has thrown their lot in with facebook and there are no "Dislike" buttons out there (to balance out all the "Like" buttons), things are only going to get worse. I suggest that all independent developers create iOS or Android apps and let WP7 slip into obscurity on its own.

LKD wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 10:49 AM

ok, where to begin.  I used to work on the XNA team.  If you know the history of who was there at the beginning, you can figure out my name.  There's a few things that need clarification.  Remember: What I say is an observation of past events as a team member.  I don't work on that team anymore, but I can make some broader statements that I'm sure are true, or close to the truth

1) Gaming the scores was always a concern, and a great deal of time was spent making sure that those kind of abuses would not negatively impact how games were rated.  This was back before ratings were allowed to be done by actual game players.  With that change, there were several considerations of how users could rate the game -- must they play all the way through to determine it sucked? Must there be a minimal time? In the end, it _appears_ that they even removed the requirement that you actually launch the game before you can rate it.  I can only (reasonably) speculate that there was a collision of priorities.  Xbox.com is a big deal, and they have a LOT of things on their plate to do.  It's doable, but priorities are almost always revenue-driven, so they simply stopped at the ability to rate a game, regardless of whether you actually looked at it

2) The whole "community games" thing was a financial failure for the Xbox org (closer to a nuclear meltdown, actually) and what you are observing is a de-prioritization of the indie game venue.  The divestiture of the XNA "brand" (and it was definitely a true brand, gained at very significant expense and legal work), the shifting of focus from Xbox to Windows Phone 7, and the currently-observed "gaming" of the ratings system points to how broken things have truly become.  While it's true that there may be team members (or "member") working on the problem, this is akin to assigning a person to shovel dirt into the Grand Canyon to fill it up.

3) Fixing Xbox.com isn't as easy as you might think.  It's generally a six-month cycle to get changes made, even important ones.  If it's causing publishers to bleed money, you'll see a change very quickly.  But indies?  Lulz.

4) The rather significant "brain drain" of the former XNA team and the sudden shoveling under the Microsoft Game Studios ("Where Good Ideas Go To Die") is a pretty strong indicator that this problem will not only not really be solved, it is likely on a "decommissioning" path (on Xbox, not WP7).  Given that indies also realize that XBLIG is a great way to spend thousands of dollars of personal time for the remote chance of making tens of dollars, the fact that the shutdown hasn't happened yet is both stunning and disappointing

Indie developers are likely wasting their time by targeting the Xbox.  It's hard enough as it is to get decent revenue from Xbox Live Arcade.  Indies should be targeting mobile.  While I personally wouldn't touch WP7 with a 10 foot debugger, each developer has to carefully consider the game concept, the audience, the distribution channel, and the ability to at least _recover_ their development investment so they can go on to building The Next Great Game.  When an indie asks these questions honestly, none of the answers involve the word "Xbox."

Oscar wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 1:36 PM

I agree that indie developers should target mobile, but I think I'll give Android a shot before WP7. Maybe in the near future, WP7 will be a better option.

Kris wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 2:27 PM

I'm not convinced WP7 is a bad market to pursue (especially if you're just porting existing XNA games). While sales of games have been bad, I'm seeing many doing very, very well on ad revenue. Yes the user base is smaller but so is the pool of competition. I also believe that unlike XBLIG, Microsoft is seriously pushing WP7 (at least for now).

It's hard for me to see XBLIG suddenly going away, many would miss it. I'm not seeing it as a very viable revenue source though and I definitely feel MS has neglected it. I hope this changes but I'm not expecting it to, which is why I'll looking at other platforms once I finish my current slate of games.

Jon wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 3:10 PM

Why don't they change the way rating works to something like the Android Market:

-You can only rate if you actually have the app

-You can only have one rating (you can change the rating, but rating it multiple times does nothing)

That seems like it would solve the problem of just getting fans to vote for a game without actually playing it and keep people from voting over and over

pwnurmom wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 3:28 PM

yeah

Dale wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 3:46 PM

Interesting story, and it's amazing after all these years a big company like Microsoft can screw up a relatively easy ratings feature, like allow only one rating per game per email address, and not verify the email address is ligit. I know there are some other tricks to pull this off, but at least prevent the easy stuff from happening.

Too bad MS is not giving the indie developer a fair shot, giving 'publishers' preferential treatment. Perhaps you start a 'publishing company' for indies to collectively submit their offerings?

Scott Beeson wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 4:25 PM

"Indie developers should target WP7 instead of Xbox"...

Okay, if my only goal was to make money, sure.  But they're two completely different things.  I don't want to develop games on a phone, I want to develop games on a TV.  This is the only way.  It's sad if they are truly letting it die.  They should fix it, instead.  Just mimic the iOS/Android Market instead of giving up.  Pathetic.

Nate wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 5:24 PM

Wouldn't a relatively easy fix be to require that in order to rate you have to give a review (like Amazon).  This coupled with standard community feedback such as "mark as spam" should sufficiently increase the "cost" of voting to the point that votes would have value.

RDRush wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 5:29 PM

Why can't MS  limit ratings to the actually game licensees as a previous entry states. Good question. In my experience the trivial oversights like this in any commercial environment makes an entity suspect on a competency level that sickens an individual. This process is so pathetic in nature as for magnitude but, the impact is shattering. It dumb founds me that these mini concerns are so merely oversight with these "professionals" and it keeps happening. Where the hell do all of these morons come from.

Mike Gambino wrote re: Microsoft Silent as Xbox Indies Marketplace Exploited
on 04-12-2011 5:46 PM

Here's what xbox should do. They should change the site to where when a person has created an account, they get only silver membership. Since they are silver, they don't have the right to rate unless they are on the xbox 360 and downloaded it personally and played it. Should keep personal history of when played. If the game isn't played but downloaded, they can't rate at all. Also, if they just went on the site, they can't rate unless they do have a preview of the game itself before knowing for a fact on how good or bad the game is.

Simple fact to be taken. And since a web designer myself, I can easily change that. Getting database settings set right, modifying the xbox settings to send information on "When Played" in the database to let the site know that the game has, indeed, has been played. Should categorize the database to know wither the member is gold of silver. Silver won't count but gold members who bought, downloaded, and play the game.

If Microsoft hires me as their problem solver on thing would be glad to help..... Besides, I have been looking for a job for 4 years. It's time to show me that someone will hire someone who hasn't had a job yet.

About The CodeBetter.Com Blog Network
CodeBetter.Com FAQ

Our Mission

Advertisers should contact Brendan

Subscribe
Google Reader or Homepage

del.icio.us CodeBetter.com Latest Items
Add to My Yahoo!
Subscribe with Bloglines
Subscribe in NewsGator Online
Subscribe with myFeedster
Add to My AOL
Furl CodeBetter.com Latest Items
Subscribe in Rojo

Member Projects
DimeCasts.Net - Derik Whittaker

Friends of Devlicio.us
Red-Gate Tools For SQL and .NET

NDepend

SlickEdit
 
SmartInspect .NET Logging
NGEDIT: ViEmu and Codekana
LiteAccounting.Com
DevExpress
Fixx
NHibernate Profiler
Unfuddle
Balsamiq Mockups
Scrumy
JetBrains - ReSharper
Umbraco
NServiceBus
RavenDb
Web Sequence Diagrams
Ducksboard<-- NEW Friend!

 



Site Copyright © 2007 CodeBetter.Com
Content Copyright Individual Bloggers

 

Community Server (Commercial Edition)