This is not an Omniture-versus-Google Analytics post. This is not a Google Analytics-bashing post.
This is a post in which I defend a decision that I helped (in some tiny way) to make when I was a Product Manager on the SiteCatalyst team at Adobe.
In 2011, businesses rely heavily on their web analytics data. Analytics may not be where we’d like to see it yet, but it certainly has come a long way over the past decade. And the more critical this data becomes, the more resistant customers will be to uncomfortable change.
SiteCatalyst 15 introduced major changes to the Omniture platform. This brought some great features with it: segmentation, breakdowns on all conversion variables, Processing Rules, etc. But it also introduced change. Specifically, it affected the way that SiteCatalyst determined the number of visits to customers’ sites/apps. In most implementations, it was a minor change, but in some cases it was noticeable in client data. (You can read all about these changes here.)
Because this platform change potentially affected things like year-over-year comparisons and conversion rate, some of our customers weren’t comfortable making the change to the new platform right away. They told us that they wanted some time to understand the new platform and its effects on their business.
As I explained on Eric Peterson’s epic (and awesome) Google+ thread last week:
The feedback that we got on this was that it was significantly painful for many users to have that change occur, but that they acknowledged the improvement. So, if you’re Adobe, do you make the change and alienate some users, or do you hold off and alienate other users?
Alyson Murphy echoed this thought:
Can you really expect Omniture to implement a major paradigm shift without alienating a ton of customers? People love their comparison data. Look at how difficult it is for some companies to shift to SiteCatalyst 15. If a relatively small change compared to what you are suggesting causes that much pain, a huge paradigm shift isn’t going to go over too well with many of their clients.
The day Alyson and I made these comments, Google announced in a blog post that it had also changed the way it calculates Visits. Now, any new traffic source encountered by a user (paid search, natural search, affiliate, social media, etc.) would instantiate a new visit/session. The reaction from customers in the blog comments has been. . . interesting:
I am see weird stuff bounce rate up 50% time on site down 75% this happened from 11th August. on most visits it count each page viewed as a new visit.
Good Grief, less then a 1% change!
“Based on our research” I would love to know how you conducted this research.
I am seeing 20% increase in visits, I thought I had finally broken free of Panda!
How I am supposed to evaluate these new metrics on steroids vs my previous metrics?
My average time on site has fallen from 7+ minutes to 12 seconds. Each visitor seems to visit the same page 6 times causing my bounce rate to be ridiculous.
I think that this update is an example of someone fixing something that wasn’t broken. Now analytics is useless.
The update makes my data virtually useless. It makes no sense.
Over the weekend, I feel that around. half of my visits are returning visitors, and the same guest may have seen the same item up to 10 times. In return, my bouncerate sky-high.
It is a vital part to have a website to have a reliable analysis program, but GA is certainly not very reliable right now, and in my case the data produced now are completely useless.
I think Google Analytics is a great tool. I use it. I use it from time to time on this blog and others, and I like it. This isn’t a complaint about Google Analytics. It is a statement about the way an upgrade which may actually be a very good thing (in terms of helping customers understand their customers and improve conversion) was handled in two different cases. I’m sure someone could explain why that last poster’s average time spent dropped precipitously, and why the new data is more accurate or more actionable.
But that’s not the point.
Conclusion? In 2011, you CANNOT just slap platform changes into your analytics platform, call it good, and expect businesses to adjust on the fly.
When I joined the Product Management team at Adobe in May 2010, we were in the midst of having this conversation with users. On one hand, it was disappointing to hear that so many felt that their users and their businesses needed time—in some cases, at companies with hundreds of users, a lot of time—to prepare for platform changes that everyone agreed were exciting. On the other, it was great to know that SiteCatalyst was that critical to various business processes even outside of the analytics team. But it’s really hard to explain to an executive why conversion rate suddenly dropped by 5% because your analytics tool changed. That’s what required time.
I’m proud that we listened to these customers and that we both a.) released a product with significant platform improvements and b.) created a system that allowed users to prepare before having these changes dropped on their plates. Is the SiteCatalyst 15 upgrade process perfect? Certainly not. But, as I mentioned above, there are considerations beyond simply the need to prepare for a change in visit calculations, and I know for a fact that Adobe continues to adapt and optimize the upgrade process.
(Also, in all fairness, Omniture has been fairly accused to making changes on the fly in the past. For example, in 2006, a SiteCatalyst point release corrected the way that search engines were identified, updating the platform to use both referring domain and referrer query string for improved accuracy. Like I said, this isn’t a tool-versus-tool argument. It’s an observation about the importance of data.)
One more thing: Anyone who tells you that only the analyst matters is fooling you. Anyone who tells you that your analytics tool only needs to serve the analyst is living in a dream world. That may have been true in 2005, but that is not how the real world works in 2011. People all over the business need data. Yeah, sometimes it’s just a perfunctory year-over-year visits comparison. Does it improve on-site conversion? Maybe not. But it matters somewhere else, to someone. Probably to someone who can influence the success of analytics in the business. Analysts had few problems with the platform changes that SiteCatalyst 15 brought, but they knew that, in order for them to succeed and to be trusted to help guide the business, their users needed to know what is going on in SiteCatalyst and not to have metrics changing unexpectedly.
So, when people say, “How come Omniture hasn’t delivered the kitchen sink yet?” remember that this isn’t a fantasy world where wholesale changes can slide right into the businesses painlessly. Google’s platform change proved that, as did the feedback we got from customers at Adobe.