If you’ve done your job, your application or website should be collecting a tremendous amount of data.
The purpose of collecting data is so you can develop smart product increments. Recall the build-measure-learn loop of product development. Your data is the learning that will fuel the next building phase. But having data isn’t enough.
On its own, data is meaningless. “It’s the secondary byproduct of other primary events that took place,” says Kurtis Williams, Chief Product Officer at Mindshare Technologies. “It’s the digital residue of a social interaction, the virtual sawdust made by an online purchase, the electronic tailings from a financial transaction.”
A study from Forrester found that while 74% of firms say they want to be data-driven, only 29% actually connect analytics to action. They’re collecting plenty of information, but they aren’t taking steps to add value to their business and product. They’re failing to turn data into insight.
The difference between data and insights
Image: luckey_sun / Flickr
Before we go further, let’s distinguish between data and insights. Data is the raw information you receive about your product’s performance, how customers are using it, or how customers feel about it. On its own, data doesn’t offer much value.
Insights, on the other hand, are actionable findings that come from data. These nuggets of information can be used to drive actual business value.
Big insights that provide a lot of value to your company can be found in small amounts of data, hidden within vast data banks. This is because the amount of data you have isn’t as relevant as the quality of the insights you pull out.
Here’s a quick example: You have a data set that shows how much time users spend using particular features of your application. You can look at it in aggregate or drill down by user. From that information, you notice that your customers use a particular feature far more than any other. This is an insight. Armed with this knowledge, you can improve your product by focusing on the development of the preferred feature.
Sometimes insights are clear. If 2,000 customers tell you a feature is broken, your insight is apparent: It’s broken and should be fixed. In many cases, however, insights are hard to identify. There is a whole field of data analysts who specialize in learning from collected data.
Insights can be so elusive that even the biggest companies with giant budgets struggle to find value in their data. In 2009, Netflix offered a $1 million prize to any group that could improve their recommendation-engine accuracy by 10%. Of more than 10,000 participants in 180 countries, the prize was only awarded to two teams. A lot of people combed through Netflix’s data, but only a few people were able to glean something actionable.
Produce insights quickly
A common mistake Agile developers (and Scrum teams) make is to spend too much time pouring over their data, looking for the insight that will create the most business value. They look for a home run scenario that will triple their revenue over night.
Generally, you should generate your insights quickly, especially in the beginning. Focus on insights that will clearly add value, not the amount of value they provide. It’s better to spend that time developing product improvements than sitting idle.
Remember that the lean approach to product development demands rapid learning. Your insights and increments don’t need to be perfect, they just need to stimulate learning. Keep your Scrum backlog full at all times and reprioritize as needed. Yes, sometimes your increments will fail, as long as you fail quickly.
How to discover good insights
Getting the most valuable insights out of your data is actually a simple process, though it takes some experience to do quickly and effectively. If you have the resources for a proper data analyst on your team, they are worth the investment just in the time they save.
Step 1: Start with the end in mind
Browsing your data is a waste of time. Insights won’t pop out at you on their own. Start with a broad goal. What do you want to improve? Do you want users to onboard faster? Do you want to improve the application’s performance? Maybe you want to improve customer retention. However, be careful that you don’t find insights that don’t really exist because you’re determined to connect the data to your goal.
Step 2: Think about context
You can’t move forward with an insight unless you understand why it’s important, so you have to compare it to other data. For example, if you notice that very few users are using your Trello integration feature, you should compare that number to other similar integration options. If triple the number are integrating with other project management tools, you should look into why the Trello integration is used less. But if each integration feature is used equally, you might decide that there’s no issue or the issue applies to all integration features. Context, in this case, is extremely important.
Step 3: Consider the insight’s alignment to KPIs
The best insights are the ones that are tied directly to your business goals because they will drive the most action. First, however, you’ll have to decide what a key performance indicator (KPI) is for your business. For instance, you may declare that product usage time is the most important KPI. Any insight you pull out of your data that would affect this metric should be prioritized at the top of your development backlog.
If you don’t designate important KPIs, you’ll run into problems where an insight’s relevance is determined by the person who found it. The backend developer will find performance insights, the UX designer will find interface insights, etc. Set important KPIs for the company as a whole and each team.
Step 4: Dig into the insight’s specifics
Once you identify a potential insight, dig deeper for a better understanding. Sometimes you’ll target an insight only to find out that its underlying data didn’t mean what you thought. For example, you might be pleased to learn that account sign ups exploded by 400% in a particular month, but upon investigating you learn that those accounts were bots set up by an automated script looking to abuse your application. In this case, the results of your investigation greatly affect how you respond to the insight.
Step 5: Use visualization for clarity
Staring at rows and columns of information is meaningless. You’ll never recognize patterns in a spreadsheet (unless the dataset is small and clear). You need to convert your data to a visual medium that helps your brain see trends.
“Tables alone are definitely not sufficient to give us an overview of a dataset,” says Gregor Aisch, data visualization expert for the New York Times. “And tables alone don’t allow us to immediately identify patterns within the data.”
Use scatterplots to map two dimensions. Use line charts for showing evolutions of data. Use bar charts to display categorical data. If you want to compare multiple groups of the same data, stack charts of the same type together. Once your data is formatted into a visualization, trends and patterns (or the lack thereof) will become apparent.
Insights are not guesses
When you present an insight to the product owner or lead developer, at some point they’ll ask you why it’s important. You should be prepared to answer this question.
Insights are not guesses or feelings. They are backed by data and lead to actionable change. If you can’t explain what has sparked the insight, then it’s not really an insight at all.