A Knowledge Gap Analysis Will Tell You What Web Analytics Can't

Web analytics are like footprints in the sand. They tell you where someone went, but not who they were or why they went that way.

Designers often place too much emphasis on website analytics to determine what changes to make to improve a website's conversion rates.

There's Only So Much You Can Learn From a Footprint

Analytics are a valuable tool to measure the effect of any design changes, but they can't tell you what changes to make.

Website analytics are reactive in that they can only be collected after you publish the changes. Using analytics as a design refinement tool forces a reactive, trial-and-error design and development process that devours time and resources. Rarely does this approach result in any dramatic improvements.

Designers are often satisfied to see a .1 percent lift, not realizing that a more proactive approach routinely produces conversion rate lifts of 2x, 5x, and even 10x.

The implicit fallacy of Web analytics is that the existing website design is somewhat correct and only needs minor refinement. The reality is that most websites are too far off the mark to be corrected with minor changes.

As they say in Maine when you ask for directions, "You can't get there from here." In such cases, Web analytics only help designers polish the brass when they could be mining for gold.

Web analytics provides incremental improvements to an existing site, but have not demonstrated the same "bang for the buck" that UX design strategies have for increasing conversion rates.

Proactive UX design processes raise conversion rates much more than reactions to analytics data that risk optimizing the wrong things by basing everything on past user behavior on poorly designed sites.

Before launching into a long-term polishing project, make sure your design is accurate, meaning that your site does the right things, first.

Knowledge Gap Analysis

One effective method to determine the accuracy of a site's design is to conduct a knowledge gap analysis. This is a straightforward process of identifying the differences between what your user personas suggest your users will actually know at each step of the task and the knowledge required to complete the task correctly.

Utilizing your user personas and task analyses to create a set of "perfect world" task flows, one for each user and task, compare your existing design to each perfect straw man task flow to identify where your design differs from these task flows.

These design gaps are the reason for your site's diminished performance. Bridging these gaps will dramatically improve your conversion rates. Once you've bridged those gaps, you'll see even more value out of your Web analytics and A/B testing efforts.

If there aren't major gaps between the perfect flow and your existing design, then you should question your analysis before simply believing that your design is that accurate. Or put another way, if your design was that accurate, you would already be seeing double-digit conversions.

Like most companies, you probably haven't done a real task analysis yet, and need to start from scratch. That's not really such a bad thing, though.

A task analysis forces you to take that objective step back and analyze your user's tasks from a different, innovative perspective, unfettered by the existing misconceptions of what your site should do. You’ll quickly realize that not only is your site incorrectly designed, but that your competitors are doing the wrong things, also. This is your opportunity to create a more task-oriented user experience that successfully differentiates your site from your competitors.

Knowledge Gap Analysis vs. Competitive Analysis

It should be noted that a knowledge gap analysis differs from a competitive analysis. Instead of comparing your site to competitor sites, you compare it to an idealized perfect task flow.

A competitive analysis produces very different results. The limitation of a competitive analysis is that it assumes the competitors have done it right, which usually is an incorrect assumption. Any analysis of an inaccurate solution will likely only yield another inaccurate design.

The greatest difficulty in performing this task analysis is avoiding the inherent biases of being too familiar with your existing design and creating task flows that simply justify your current site. Remember, your design is inaccurate, so if your analysis says it isn't, then your analysis is equally as inaccurate.

How to Include Knowledge in Your Design

The magical and innovative part of the knowledge gap analysis process is that it not only identifies the gap, but also suggests what knowledge you need to add to your design.

Your task analysis should indicate what knowledge is required by the user to successfully navigate your site and perform necessary functions. Your design must somehow include that knowledge if you can't expect your users to already have that requisite knowledge.

Some common ways to include knowledge in your design are to promote a single best practice approach to completing a task, provide templates, and provide "calculators" to help users more accurately determine what they need.

If there is more than one way to perform a task, avoid relying on the users to know what the right way is.

When an engineer sees 100 users doing the same thing 100 different ways, they think that they need to design an interface flexible enough to allow anyone to perform the task any way they want.

When a UX designers sees 100 users doing the same thing 100 different ways, they realize that no one knows how to do it the right way and they need an interface that guides them down a single, best practices path.

While conventional thinking might suggest a flexible interface, a single path interface will likely be much more successful.

Provide Templates or Calculators

If the user’s task involves creating something, avoid forcing them to create something from scratch. Give them reasonable templates that they can modify to suit their needs.

It's far easier to recognize that a template needs changing than it is to create the correct product from scratch. A template will likely contain elements that many users had not thought of until they saw them in the template. FedEx Office Print Online provides templates that users start with and then walks the users through the potential changes, eventually arriving a just the right result.

"Calculator" is more of a metaphor than an actual calculator. For instance, an insurance site might ask users a few questions to determine what insurance needs that user might have.

Does the user have kids? Are the kids active in various sports? Are they nearing graduation?

These questions determine what type and amounts of insurance a homeowner might need and how those needs could change over time.

If you’ve ever had to change your insurance policy, you’ll recall that insurance companies each offer different policies and options that can be quite confusing to the average consumer. Given the complexities of insurance policies, relying on the user to assimilate and understand all of these complexities in order to choose the right options places an inordinate reliance on user knowledge.

Since we know that individual user knowledge is highly variable, any design that relies on that highly variable user knowledge can only achieve an equally variable level of successful. A calculator reduces the dependency on individual user knowledge and increases user engagement, which increases conversions.

Summary

Conducting a knowledge gap analysis should be your first step in finding ways to increase your conversions. Once you've corrected your site design, your Web analytics will provide more accurate insight into what incremental changes you could make to refine your conversion success.

Don't be afraid if your site looks and acts differently than the competitor sites. It should. If it doesn't, then either they have done a knowledge gap analysis and you are just now catching up, or you've done something wrong and are vulnerable to someone else doing a knowledge gap analysis and leaving you behind. In either case, you can't afford not to do it.

New processes and tools are evolving everyday in the fast-paced world of e-commerce and this is one tool you should definitely add to your website analysis toolkit.