Internet Math - Held to a Higher Standard
Today’s Wall Street Journal has good coverage of the long-simmering debate between Comscore and Nielsen with regard to ranking large Web sites’ popularity and total unique visitors.
According to the article, Comscore shows ~20% higher unique visitor counts for many sites, apparently do to differences in panel composition. For example, Comscore shows 11.977 million visitors in March for AOL Body. Nielsen shows 9.962 for the same site and date range.
It’s a puzzlement that is almost as old as the Web itself. Jupiter/Media Metrix data used to conflict with other services, and today’s WSJ article didn’t even touch on data from Compete, Hitwise, and other services that today purport to actually be more accurate than Comscore/Nielsen due to their methodology.
I find this line of inquiry amusing. The only reason people fret about data discrepancies online is because there are multiple services. You don’t see widespread questioning of the accuracy of Nielsen TV ratings (other than DVR issues), and you don’t see much concern over the accuracy of Arbitron for radio. Why? Because they don’t have competitors trumpeting a better mousetrap.
I believe it to be true that there are 450 Nielsen TV households for Greater Phoenix, a metro area of 3+ million people. That’s inevitably going to result in statistical abnormality and fuzzy math. I would argue that while Comscore and Nielsen are indeed a ways off in their comparative numbers, the overall data certainty online is better than offline across-the-board.
Part of the problem with data discrepancies is what defines a “visit”. There are MASSIVE results differences when you take a given Web site and analyze it’s traffic using the leading Web analytics packages like Google Analytics, Omniture, Webtrends, CoreMetrics et al. Each “counts” visits slightly differently, and this problem is getting worse, not better with the increased use of AJAX and other programming tactics that push data at visitors without a “page view”. The way I see it, if super-techy companies like Google and Omniture can’t get their numbers to align in the slightest using actual Web server data, to expect a sample based service like Comscore or Nielsen to align is unrealistic.
