Archive for SaaS

Testing and Enterprise Software

A few weeks ago I asked a question on Twitter, looking for feedback about a certain workflow in the product I help to manage. In part because the product in question (Adobe Analytics) caters to an audience that thinks about data-driven optimization, a well-meaning user of the product suggested that we A/B test the workflow with users, and then promote the “winner” so that it becomes the workflow for everyone.

A/B Testing Is Almost Always a Great Idea

Generally, I am a huge fan of A/B (or “split”) testing. In fact, my company sells an industry-leading testing and optimization solution (Adobe Target) which helps companies all over the world improve web sites and mobile apps for their customers. While I myself am not an expert on testing theory, the basics are easy to grasp: If you’re not sure how to improve your customer experience (and who is ever really sure?), run a test and let the results speak for themselves. It’s an awesome concept, the returns on investment are huge, and almost everyone who has a digital presence should be doing it.


But Probably Not for Enterprise Software

The amazing Blair Reeves has written an excellent piece on the difference between product management for enterprise software and product management for consumer software. I highly suggest that you read his post, because it gives a lot of context for my opinion on this matter, and I’m not going to restate it all here. However, I will quote one particularly salient point:

When big companies pay you millions of dollars for software, the last thing they want is major, unannounced and unexpected changes to the product . . . This is even more true for business-critical applications like the ones I’ve spent my entire technology career working on, like digital analytics, marketing intelligence and ecommerce platforms. If one of those goes down, it’s not just annoying — it’s lost business. The stakes of failure are potentially huge, and not just because customers expect the product they paid for to always be available.

The reason A/B testing is great for most web sites but usually terrible within enterprise software products is that enterprise users (“pay[ing] you millions of dollars for software”) do not like to be jerked around when it comes to their user experience. I mean, nobody likes when things suddenly change on them, but the whole relationship is different when your customers are massive organizations rather than individual customers. Many of the customers that I personally work with insist on months of advance notice when there are user experience updates. Your customers learn, and they train their (often very large) organizations on how to use your tool. When you change it on them in the name of testing, they are often left hampered in their ability to navigate to the key workflows that provide value.

From a testing perspective, the reason this doesn’t work is that test participants aren’t supposed to know that they are in a test. (More on that later.) Let’s consider a new user of your tool. We’ll call him Charles, and he’s just gotten access to your product. You want Charles to have a positive experience, since he is part of company paying a lot of money for your software, and the tasks he completes with your product may well contribute to the customer’s view of their return on investment with your company. To get familiar, Charles watches a few training videos recorded by your education team. He reads a bit of documentation as well.

Now Charles logs in. But wait! This doesn’t look quite like the videos. Where is the feature he read about in the documentation? This doesn’t look right! Charles is confused, and frustrated. To make matters weirder, Charles asks a friend at a different company, Amanda, why the product doesn’t match the documentation or videos. But Amanda sends Charles a screen shot: to her, the product looks exactly what is documented.

What happened here? Charles ended up in a certain test group, getting a different variant of the user experience than other users. Amanda was in the control group, which got the traditional user experience. There could be multiple test groups, each with a different experience. Some of these experiences may indeed be better than the control, and users may be able to find their way painlessly. But many of them won’t be, and frustration will mount. Nobody is actually going to quit Facebook because their News Feed looks different than someone else’s News Feed (and even if a few people did, it wouldn’t put a dent in Facebook’s overall growth), but they often will abandon your software, putting revenue at risk for you.

Why not simply create separate training, documentation, etc. for each test variant? This certainly would help, but show me the software company that has enough resources and coordination to support creating multiple versions of all supporting materials and I’ll show you the Lost City of Atlantis.

I may not have even touched on the biggest reason to avoid A/B testing inside of enterprise software products: the duplication of effort for your user experience design and product development teams. Instead of building each feature once, they have to build it 3-4 times. The opportunity cost of approaching workflow changes this way is huge, and puts a ton of burden on your most valuable resources.

(Another potential issue here is sales. The last thing you want during a demo to a prospect is an unfamiliar experience that your sales team doesn’t know how to navigate. This can be mitigated by excluding demo accounts or demo environments from tests, but is still risky at best; when the customer buys, what they see may not match what was demoed.)

How Should Enterprise Software Perform Tests? 

I’m a bit worried that this comes across as if I am saying that product managers should simply take their best guess on a user experience. Far from it. The answer, however, is prototyping and user testing.

(None of this, by the way, is intended to say that enterprise software developers can’t programmatically test the size of a button or the content of a hero banner. My warning is to avoid A/B testing in production on key user workflows.)

Prototyping involves user experience designers and/or developers creating a light version of the proposed change. This can be in a development or beta environment, or can even (in the case of UI updates on the web) be a JavaScript bookmarklet that appears to change the layout and CSS in your product for a user, without actually making any underlying code changes.

User testing involves working with a small group of users, typically of varying experience levels, and possibly in different industries, to see how well they understand the prototype. These tests should likely be run by your user experience design team, with product management and development supporting. The test administrator should give the subject a series of tasks to complete given the proposed new experience. Their ability (or inability) to adapt and intuit their way through the new experience, while talking you through their thought processes, will yield valuable insights.

In some cases, you may even be able to “A/B test an experience in production” by explicitly inviting users to join a test. This would drop them into a variant on their normal experience, and they can provide feedback via a form.

Prototyping and user testing early in the software development process will not only ensure that the experience you’re building is easily understood, but it will allow all of the supporting materials (documentation, training, etc.) to be created around a polished final product, so that there is harmony across all of the “channels” through which users master your product.

So my advice is to test away, but do it in the right way for the enterprise. Involve your customers in testing prior to (and during) product development. But don’t risk alienating your users, whose experiences pave the road to your future earnings. Don’t give them unexpected variants on an experience that their business relies on.

I got 99 problems . . . but are they customer problems?

A lot has been written about “user versus customer” as an abstract concept ever since (and probably long before) Jack Dorsey asked Twitter to stop calling its users users and start calling them customers. I’ve also seen product managers and UX designers write about how “product owns the customer; UX owns the user.” I think most of this discussion is useful and relevant in defining roles and responsibilities in a product organization, and I’ve referred to various aspects of it many times in my own work with other product managers, UX designers, and developers.

That said, while I don’t spend a lot of time on product management blogs, I haven’t seen a good, concise explanation of how to think about user problems versus customer problems when defining product requirements for design and engineering. I’m writing this post mostly to remind myself to think a little differently in the new year.

I often find myself sitting down to write up a PRD and starting from what I call the “customer problem.” But is it really the customer problem? For example:

  • The customer needs to better understand retention and churn in their own customer base.
  • The customer needs a better way to visualize the overlap between marketing segments.
  • The customer needs to be able to upload massive amounts of customer data for analysis in our tool.

No. No. NO. These statements do represent problems, and they are probably even worth documenting and elucidating as we present our ideas and try to win advocates and acolytes to our cause, but they are not customer problems. They are user problems. They describe what a user needs to be able to do once they are already in the product.

Okay, then; what is a customer problem?

A customer problem—which should be at the beginning or near the beginning of every PRD and concept deck we create—should describe how the customer (a business) is failing to grow. Both customer problems and user problems describe inefficiencies, but customer problems come with a clear indication of the actual (monetary) value that a solution would provide to the businesses we serve, while user problems may only hint at that value.

As such, customer problems are more likely to describe something that would keep the CEO or CMO or CIO up at night than they are to describe something that keeps the day to day user of the product up at night. That’s a good thing, because it can’t hurt to start speaking the C-suite’s language.

Here are a handful of examples of what I mean:

  • The customer has reached a point of diminishing returns from their acquisition budget but still needs to hit a top-line growth target of 20% YoY. The most efficient way for them to do this is by personalizing the on-site experience.
  • The customer is facing intense downward pricing pressure from a smaller competitor who is trying to steal market share, and they are looking for new ways to engage with their own customers to keep them from attriting.
  • The customer knows that segmentation is everything in marketing, but beyond their legacy market segments they have been going after for years, they do not know how to find the right groupings of individuals so that they can reach them in the right way at the right time.
  • The customer has been unable to consolidate customer data stores across the enterprise and, as such, has not been able to empower various marketing teams with insights at the brand level.
  • The customer has millions of dollars flowing into marketing campaigns across various channels, but has only a very limited view of how those dollars influence not just short-term revenue, but long-term customer value.

As I’m sure you can see, user problems can/should flow from customer problems. But we could solve any of the user problems a few paragraphs up from here without actually ensuring that we’re solving the real underlying customer problem that will help users of your product grow their business.

As I’m sure you can also see, the customer problems I’ve listed here are composites, more indicative of a market problem than anything any one particular customer is facing uniquely. (Unless you’re building a product for an individual customer, you will want to pick a someone from across your customer base to represent the customer problem, but it should reflect broader trends, obviously.)

All of this might seem obvious, in which case maybe you haven’t made it to this point in the post. In a way, it is obvious; we all probably try to focus on customer problems, but we often settle for user problems even though we know better. Why is that? As a colleague pointed out, it’s probably because we spend a lot of timing documenting problems, use cases, and solutions for developers, who do solve user problems with their work; thus, we naturally tend to put things in terms that developers are most likely to understand and relate to; however, those user problems should still be grounded in customer problems. This will enable engineering teams to understand the deeper problems as well, giving them the best chance to innovate in ways you could never have envisioned.

What if I don’t know what the customer problem is?

As a product manager you are probably fairly close to your users. Your users sit within the organizations that pay for your product, and many of them likely understand the factors that are limiting growth of the business. If you ask them what challenges the business as a whole is facing, they may or may not get right to the heart of the matter, but this is why (no pun intended) I love the “Five Whys” method of interviewing.

The theory is that within (roughly) five iterations of asking a person “why?” you can get to the root cause behind the initial problem described. You ask what problems are facing the business, and they give answer. You ask why that’s a problem. They answer. You ask why that’s a problem. They answer. Lather, rinse, repeat. By the fifth “why,” you should be at a root problem that motivates the users you’re interviewing to action in a big way. That should either be the customer problem or very close to it.

You may need to ask to interview the user’s manager, or his/her manager’s manager in order to get the level of detail you want in order to truly understand the customer problem, but the “Five Whys” method will help you get there.

A final huge benefit to thinking about customer problems

One of the most underrated skills in product management, in my limited experience, is the ability to convince people. You have a vision for your area of product ownership, and you’re likely competing for resources (development, UX, etc.) with other products or other features. You will get nowhere with that vision unless you are able to convince key stakeholders that what you want to do is supremely valuable to your customers (and therefore to your own business).

There are many factors that play into the “convincingness” of a product proposal, but in my (again, limited) experience, proposals that are based in a rock-solid knowledge of the problems your customers at large are facing, where the product manager can speak knowledgeably and authoritatively about the business problems preventing customers from growing, tend to convince more people more completely. These projects tend to get funded because stakeholders feel the customer’s frustration much more acutely than if the problem presented is purely a user-level problem.

Think customer

One of my goals for 2016 and beyond is to do a better job in interviews with users getting to the root of the customer problem and being deliberate about making that the central focus of my product requirement/design work. (Incidentally, I think I’ve been reasonably successful at this, but more by chance than by choice.) I have no doubt that it will produce better products and features; I just wish I had come to these realizations much sooner.

Ah, well; I never said I was a fast learner.