These common issues can torpedo your reporting and analytics initiatives. Make your data work harder and generate higher-quality insights by avoiding these key mistakes.
Without a defined set of business objectives, your analytics efforts are going to struggle. Before diving into your data blindly, ensure that you have gone through a structured process to:
The result of this exercise will provide a framework that orients all of your analysis to align with business outcomes.
An overall trend can be deceptive. Consider the following results that were recently brought to us by a digital marketing manager after a creative refresh of their paid social ads:
The creative refresh had been heralded a great success with an impressive 18% increase in click-through rate – but two mistakes had been made.
The first mistake was not conducting a randomized A/B test. The second mistake was that the data had been completely misread, with the marketing manager falling prey to Simpson’s Paradox.
Examining the data again, broken out by audience and layering in the impressions per audience, generates a much different insight:
From this view, we can clearly see that the new creative underperformed in every audience and that the top-line lift was due to a dramatic shift in impressions across these audiences.
Make sure that you don’t misread that top-line trend and always look at your key data segments for context.
One of the data community’s most-used maxims – and for good reason. Understanding what is the driving cause of consumer behavior is a goal of almost any marketing or customer analytics practice. To keep it easy, we follow a simple process at Definition 6:
Generate hypotheses from correlation, and test to conclude causation.
A well-designed and randomized A/B/n test is one of the safest methods of taking insights to the next level.
Image from xkcd
The human mind is an amazing tool. One of its more remarkable gifts is the ability to recognize patterns and apply those patterns to future scenarios. In data analysis, this can be both a gift and a curse. Our fundamental need to find patterns and apply them can lead to the misuse of data.
For example: Take an A/B test that is run in a single market or geographic area, such as a city. If a winner is declared in that city, does that mean it will win nationally? Too often this type of result is overgeneralized and acted on with unexpected outcomes.
In this case, an expanded test that touches on a diverse set of markets could provide a much more powerful and generalizable test result.
With every insight your analytics program makes, they should also be asking themselves a question: How generalizable is this insight?
Messy data has made a victim of us all at one point or another. Defend your business against catastrophic data failure by comparing insights and results from multiple data sources.
This is easier said than done, of course – but it still should be part of your measurement strategy. Periodic checks and balances of your data are vital to ensuring data quality and hygiene remains at a high level.
Setting targets for each one of your KPIs is a crucial exercise that ensures 2 things:
This simple exercise can radically change the conversation in your reviews with business leaders. Setting targets can end the time wasted discussing “are we succeeding?” and focus on what is driving success.
Data is a complicated and endlessly nuanced subject – while avoiding these six mistakes is a good starting point, there is always more to learn.