SPEAKER_00 0:00

Welcome to Cavebits, your source for the latest insights in website behavior analytics brought to you by Mouseflow. In this episode, we'll talk about how CRO experts design hypotheses for experimenting. Let's get started. So, we all do it. Cling to gut feelings and personal theories, even when making choices for our businesses. But when it comes to conversion rate optimization, or CRO, relying on hunches can be disastrous. It can lead to poorly designed experiments that miss the mark and fail to deliver real results. By embracing a data-driven approach to CRO, you can ditch the guesswork and see what a data-driven CRO hypothesis can deliver. In this episode, I'll guide you through developing data-driven hypotheses that set your CRO tests up for success. You'll also learn how to use data effectively, identify real conversion problems, and build hypotheses that drive real results. What is a data-driven hypothesis in CRO? A hypothesis in CRO, and pretty much everywhere else, is a testable statement that looks as follows. I quote, we hypothesize that if we introduce a certain condition, something that we want to try changing, then a certain metric, sometimes called a primary or a North Star metric for the hypothesis, will change by a certain percentage. A data-driven hypothesis adds one very important part to this formula. I quote, we hypothesize that if we introduce a certain condition, then a certain metric will change by a certain percentage, because the data we have suggests that it will impact user behavior in a certain way. It's based on real data and insights rather than guesswork or somebody else's gut feeling. I'm pretty sure you've noticed that some people suggest simply increasing the button size or color to solve every funnel leak they see on GA4. But why are they so sure that it's all about the button size? Where's the data that indicates the users don't notice the button or get distracted by, say, a bigger button? Without such data, that's just guesswork. Manuel DaCosta, a CRO specialist, sums it up nicely that CRO isn't just about talking, it's about taking action based on a data-driven solid plan. DaCosta says, share your experiments, get them excited, make it visible, build a culture of experiment. This is the experimentation CRO equivalent of eat healthy and do some exercise. It's what I call TBU, true but useless information. It doesn't give you much to go off, all talk, no action. Then, what could a data-driven hypothesis be? Here's an example. If we reduce the number of form fields in our checkout process, we expect to decrease the cart abandonment rate and increase the conversions, because our data from Google Analytics shows that there's a 40% drop-off rate at this stage, and our form analytics suggest that users struggle with fields such as address, second line, and a landline phone number. The data from our analytics show that our current cart abandonment rate is 65% and our average conversion rate is 2.5%. We expect to reduce the former by 15% and increase the latter by 5% by simplifying the forms. So what data can be used to inform a hypothesis? There are plenty of data sources to use as support for your hypothesis. The more different types of data sources you can use, the better. First up, quantitative analytics. Quantifying user behavior via website analytics like GA4 or Adobe Analytics, or conversion funnel analysis, like what Mouseflow offers with its conversion funnel optimization tool, is a valuable starting point for hypothesis development. But solely relying on quantitative data can lead to incomplete hypotheses. You see the data, but you don't see the underlying reasons behind it. Without this information, it's difficult to interpret user motivations and underlying causes of website or product issues. Here's what I mean. Imagine you're an e-commerce store owner and you notice that a considerable number of people are abandoning their carts at the shipping page. Quantitative data might tell you that the average cart abandonment rate is 50% at this stage, but this doesn't explain why people are leaving. To get a better understanding, you can observe how users interact with the shipping page, and you might discover that, perhaps, the shipping options are confusing for them. Now, that's already qualitative data. Second, behavior analytics. In the example I just mentioned, diving into user behavior data allowed you to find out the reasons behind the high car abandonment rate. Now you can formulate your hypothesis. It's gonna sound more or less like this. We hypothesize that if we change the description of shipping options, we will decrease car abandonment by 10%, because Google Analytics data tells us that there's a 50% drop-off at this stage, and our movement heat map data shows that our shipping options are confusing. And then you can proceed to documenting and running your experiment and make targeted changes to improve the checkout process. While quantitative data can help you identify areas of improvement, qualitative data is important for understanding the underlying reasons. To collect qualitative data, one of the tools that you need is a behavior analytics tool like MouseFlow. Behavior analytics platforms offer different ways to analyze user behavior, with the most well-known ones being website heat map tools and session recording tools. Heatmaps visualize user clicks, mouse movements, and scrolling behavior, helping you see which areas of your website or app attract the most attention. Session recordings capture actual user interactions on your website or app, allowing you to observe their behavior as if you've been sitting right next to them at the computer while they were browsing your website. This can be especially useful for identifying usability issues, such as confusing navigation, unclear instructions, or unexpected errors. Now here's the thing I believe many overlook. Session recordings can be effective in persuading stakeholders to make changes. I'm going to share a quote by Matt Scabesbrook, a director of optimization at WeTeach CRO, who said, Nothing convinces stakeholders better than a video demonstration of users failing epically on your website. Up third, user feedback. User feedback is another valuable source of information for hypothesis development. I guess that's one of the reasons behavior analytics platforms offer feedback survey tools as one of their features. It lets you hear unfiltered opinions and suggestions from real people using your product or your website. User feedback can explain why users like or dislike certain aspects of it, providing context for their actions. In case you're wondering why to bother with running user feedback surveys, the answer is that neither quantitative data nor session replay or heat maps can capture the same depth of understanding as user feedback. User feedback allows you to hear directly from users, so it provides a more personal perspective that can help you identify areas for improvement. And also, it can serve as an additional data source to support your hypothesis. The more correlating sources you have, the higher the probability that your hypothesis will be true or at least will reveal something interesting. Fourth, customer conversations. Quantitative data reveals what happens on the website, qualitative analytics shows you the why, and talking to customers reveals their thoughts and intentions that made them act this or that way. Well, although I see everybody talking about customer insights or conversations, many teams still struggle to extract their full potential. Navigating customer conversations effectively requires more than just asking questions. I have a few suggestions for running customer conversations. Here are three suggestions we have for making the most out of customer conversations. Don't dominate the conversation. Customers easily agree, but you don't want them to support your opinion. You're talking to them to hear theirs. So instead of saying, Did you find the checkout process easy? ask, can you walk me through your experience using the checkout? Highlighting any areas that felt smooth or frustrating. This makes the conversation much more open-ended. Don't settle for easy answers. Instead of what did you think of the product, ask, can you describe your initial impression of the product? How did it make you feel? This is much more emotion focused. And also, don't let insights get lost in translation. You need to bridge the gap between a conversation and action. Conclude by asking, based on your experience, what changes would make the product easier to use for you, which is much more action-oriented. Fabian Hans, a conversion consultant and founder of Cognateer, shares some great advice. He says, pay close attention to users' concerns, objections, and open questions during the buying process. It is also crucial to understand the user's buying journey and decision-making process by employing methods such as active listening in one-on-one interviews, group interviews, or brainstorming sessions. Rapid user tests and on-page surveys, when connected to user session recordings and click maps, can further generate valuable insights. So, those are the four main sources of data for formulating hypotheses. If you get an experiment idea that you think is really great from one of the sources, try using the other data sources to prove or disprove it. And don't worry if your idea gets disproved. As you go through the data, it's likely you'll generate ideas for a dozen more hypotheses. If you want to get more inspiration and learn how renowned CRO professionals come up with their hypotheses and learn from their experiments, check out our CRO journal where they share their experiences. Now that you have created a great data-driven hypothesis, it's time to design your experiment. Here are three key steps to ensure your experiment is well designed. Document the experiment. Clear documentation ensures everyone involved stays aligned, facilitates future analysis, and lays the groundwork for a successful implementation. Identify the KPIs you'll measure, aligning them directly with your hypothesis. Are you tracking conversion rates, engagement metrics, or specific button clicks? Choose tools suited to each KPI. For example, Mouseflow for heat maps or Google Analytics for page views. Define the anticipated results for each variation. Don't just guess. Use A B testing calculators or historical data to set realistic expectations. Determine the optimal testing duration based on traffic volume and expected effect size. Too short and you risk inconclusive results. Too long and you waste valuable resources. Most A-B testing tools have built-in calculators that automatically calculate the required duration based on the amount of traffic the page receives. And document the insights gained from each experimentation cycle and the iterations or refinements made to the experimental design. Capture both quantitative metrics and qualitative observations, including unexpected findings and emerging patterns. Next, analyze the results for insight. It may sound strange, but regardless of whether the result was clear-cut or not, or if it was a success or a failure, what's more important is what you've learned from it. So focus on understanding why your hypothesis was proven or disproven. Then, make sure you use post-experiment segmentation to find out if some groups of users exhibited different behaviors. Here's some ideas for you. Desktop versus mobile visitors. That's the most obvious one, but it may reveal important differences in behavior between desktop and mobile users that could have influenced the results. Persona-driven analysis. Analyze results through the lens of your established user personas. Did a specific variation resonate particularly well with a specific persona? This can inform targeted optimization efforts. First time vs. repeat purchasers. Analyze how each group interacts with your site, identifying pain points for first-time users and loyalty drivers for repeat customers. High value versus low value customers. Understand the browsing and buying behaviors of your most valuable customers to optimize the experience for them, while also identifying opportunities to convert low value customers into high value ones. Complimentary product buyers. Analyze users who purchase specific product combinations to identify cross-selling and upselling opportunities. Wish list users. Analyze users who add products to their wish lists but don't purchase. See if you can offer incentives or personalized recommendations to convert them. Promotional channel. Analyze user behavior based on the channel they receive the promotion through, such as email, social media, or whatever else. This helps you understand which channels drive the most conversions and tailor future promotions accordingly. Use insights to inform future experiments. The learning never stops, right? Think about your findings to continuously refine your approach. Analyze results to identify areas for further exploration. Did a specific element within a variation show promise? Explore it further in a new experiment that you add to your CRO roadmap. I'd recommend that you allocate resources based on learning and focus on areas with the highest potential impact based on data-driven insights. And to conclude on data-driven CRO hypotheses. Diving into all this data may sound all too complicated, but effective CRO isn't about quick fixes or lucky guesses. By following the steps outlined in this episode, you'll no longer be stuck in a cycle of underperforming tests. Instead, you'll develop data driven hypotheses that fuel targeted experiments, show you hidden conversion opportunities, and pave the way for sustained optimization.