In an ideal world there is both valuable quantitative as well as qualitative data available to you.
You can’t say that one data source is better than the other. They complement each other and provide you with a more accurate picture of what’s going on and why.
Both data sources are very helpful in the field of conversion optimization.
Well thought out hypothesis – based on quantitative and qualitative data – are important to define the best A/B test experiments.
However, it is very important to understand the limitations of qualitative data analysis.
In this article I share six common problems with qualitative data that you should know.
Sampling-Related Problems
The first three limitations are sampling-related issues.
1. Limited Sample Size
Contrary to quantitative data where you often have a great amount of data available, is sample size one of the challenges of qualitative data.
If you browse on the internet, you find out there is no general agreement on the ideal sample size for qualitative research.
It is very costly to perform extensive qualitative research with hundreds of participants.
And is it really needed to question so many people to get valuable insights?
Watch this video to get a better understanding of this topic:
Two tips about your sample size:
- Rule of thumb: you need more participants if new participants keep on providing you with relevant, new insights.
- Be flexible; don’t rigidly set the number of participants at the start.
2. Sampling Bias
Sampling bias definition by Wikipedia:
“In statistics, sampling bias is a bias in which a sample is collected in such a way that some members of the intended population are less likely to be included than others.”
In other words, your qualitative sample will never include a representative overview of all the different people that come to your website.
It’s important to keep that in mind when interpreting test results.
3. Self-Selection Bias
Do you like to participate in surveys? A few of you might say “Yes” and others think “Arghhhh”.
This is the exact problem here. It’s a free choice to participate in a research study or not.
On the other side, quantitative data is gathered from most people whether they like it or not.
Just sign up for Hotjar, set up a heatmap and the data will be collected for you.
Ok, I don’t talk about the tech-savvy people here. ;-)
Sampling and self-selection biases are closely related and limit the usefulness of qualitative data.
Observation Biases
The second group of problems with qualitative data include observational biases.
4. Hawthorne Effect
The Hawthorne Effect can best be described as:
“Participants in behavioral studies change their behavior or performance in response to being observed.”
For example, your opinion about a particular website might be different when you know you are being observed if compared to when you (don’t know) you are being observed.
I recommend to watch this video (it clearly explains the Hawthorne Effect and its background):
5. Observer-Expectancy Effect
Let’s say you are running a survey and function as an observer in the research room. You are walking around and observe the participants.
Do you think you won’t influence the results?
It is known that researcher’s beliefs or expectations causes him or her to unconsciously influence the participants of an experiment. This is called the observer-expectancy effect.
6. Artificial Scenario
Most experiments include pre-set goals in a specific environment. And you can’t get feedback on things you don’t ask.
For example, you run an experiment for an ecommerce website.
Your goal is to find out whether the form (where people leave their personal information) functions well or if anything needs to be improved.
In this case it is such a focused goal so that you won’t learn about other valuable things through this study.
The participant might have a lot of other things to say, but without asking them you won’t know it.
Conclusions
As you can see, there are a many challenges with qualitative data.
However, marketers can perform extremely well if they use this data in combination with quantitative data to form strong A/B test hypothesis.
Refrain from changing your website on just a small set of qualitative responses.
Instead, enrich your conversion optimization framework with all data sources that are available to you and get more out of your testing efforts.
What’s your experience with qualitative data? Do you use it in combination with quantitative data?
One last thing... Make sure to get my automated Google Analytics 4 Audit Tool. It contains 30 key health checks on the GA4 Setup.
Leave a Reply