» How Would You Even Know If the Analytics Were Correct?

How Would You Even Know If the Analytics Were Correct?

May 11th, 2008 by Greg Harris

I’m sitting here with 3 LCD screens in front of me with database queries, spreadsheets and browser windows open trying to determine how accurate Mobilytics is.

Let’s forget about mobile for a minute and just talk about web analytics in general. Before factoring in the complexities of mobile, I wanted to get a reference point where I could say that Mobilytics was correctly tracking a regular Internet web site.

Since we are not using JavaScript, this would be a great test of the accuracy of our server side snippet code in general.

So I decided to compare Mobilytics to a few web analytics products out there which I will not name. I will say that one of them is Google Analytics though. The problem with Google is that I can’t drill down to individual visits to compare page view by page view.

So here’s what I came up with. NO WEB ANALYTICS PRODUCT WILL EVER BE RIGHT!

It’s just not possible. There are too many opportunities for minor issues to impact the data. The real question is, how would you know if it was right anyway? And how good is good enough?

Here I am on a Sunday morning comparing my raw log files and database queries side by side with 2 other web analytics products. I placed each product’s tracking code on the same Internet web site, so you would expect them to match up.

Not the case. And I’m not just talking about general numbers and percentages. I am auditing individual page views one at a time. While I can understand the occasional duplicate appearing, or the time on the page being off, how is it that I have visitors in my data that don’t show up at all in this other vendor’s product? The one particular visit that led me to blog about this is coming from the U.S. on CabelVision using Firefox. So why would it not be appearing in the other product?

One thing I can think of is that Web Analytics rely on JavaScript and Cookies. Maybe this visitor turned them off? Well since we are using server side tracking code, we have the complete HTTP headers and other info so I can easily check. (doing this now), be right back…

Ok so here’s what I found. Clear evidence of a visitor who came through a referral link on a blog, and is not a crawler or search bot of any type. While I can’t tell in this case if they have cookies turned off, you would think with Firefox that would not be the case. But let’s assume it is. Wouldn’t we at least register a visit? Not only did I lose a visitor, I lost a referrer, a page view, and I would never have even known. What if I am an advertiser and I am paying to attract visitors? Is it ok for one to be completely missing?

Who else is reviewing logs and comparing vendors to see how accurate they are? Now I wonder how often this happens? I’m only reviewing a 1 hour period on a single day.

On mobile we can’t rely on cookies and Javascript, why does the wired web take them for granted? I guess the same way we need to discard data sometimes in order to maintain the integrity of the data, the web vendors do also. It just seems to me that there is no reason you can’t use an image tag and Javascript.

And just to be clear, I don’t even know if this is the reason the visitor was missing. I will be looking further though.

So the simple conclusion I have drawn from hours of comparisons in general is that putting the numbers side by side will NEVER match up exactly. At some point they will vary. But there’s no real way the average user would know anyway.

I’m looking forward to putting a few of the mobile products side by side and seeing how we stack up. The quality and accuracy of the reporting is the single most important thing we are focusing on, and that is what will make Mobilytics the leading third party analytics and tracking solution for mobile content. Why else would I be comparing single page views and HTTP headers on a beautiful Spring Sunday?