Mouseflow Podcast: The show for marketers who obsess over results.
Converting Talks by Mouseflow — real conversations with the marketers, analysts, and practitioners obsessed with turning clicks into customers. No fluff, no theory — just honest talk about what's actually working in conversion, UX, and digital growth.
Mouseflow Podcast: The show for marketers who obsess over results.
"More Tests, More Winners" with Ruben de Boer on what it actually takes to build an experimentation culture
Most teams think AI will fix their experimentation program. Ruben de Boer disagrees, politely but firmly.
Ruben is a freelance CRO and experimentation consultant with 15+ years of experience, clients like Vodafone, eBay, and Rabobank, and over 27,000 students on Udemy. In this episode, recorded in Amsterdam, he talks through what AI actually changes in CRO (and what it doesn't), why psychology should guide every experiment from the start, and what experimentation maturity really means, based on a report he co-published with 15 industry leaders.
Also: how to know when an AB test is actually done, and why the answer is more precise than most people expect.
Converting Talks by Mouseflow, conversations that convert.
Follow the clicks and see the flow. Where do your users really go? Tune in right now and get in the zone. The Mouseflow podcast is on. Yeah. Let's go. Hi, and welcome to Mouseflow podcast, Converting Talks. And today, I have Ruben the Boer. Yeah. Oh, very good. Very pronounce it correctly on a Dutch way. Yes. Yes. Good that I asked you just, like, two seconds before I was about to say the bow. But so it's like literally converting talks. It couldn't be even more into the point with what you do and what you basically talk about and speak about. For those who don't know Ruben, he will have the chance to introduce himself in a sec. But Ruben, I would say, is COO guru, experimentation guru, in my book at least. But I also know that in many others minds. And I'm so lucky actually to be in Amsterdam today, and had the opportunity to meet up with you in a cozy Amsterdam area. Yeah. so that's I've been really looking forward to talk to you but before we kind of deep dive a little bit about experimentation, AB testing talk and converting talks, tell me a little bit about yourself. Yep. Thank you for the nice introduction. And I was going say thank you for having me, but you're actually in my place. So now for those who don't know me, my name is Ruben. Yes, I live in Amsterdam. I've been in the business of CRO optimization experimentation for over fifteen years now. And today I'm a freelance CRO and experimentation consultant, and I still have a couple of courses on Udemy, a couple of CRO courses, online psychology courses where I have over 27,000 students worldwide, but really enjoy this line of work, do it with amazing clients, with amazing teams. yeah, happy to be here. Yeah, good that you're happy to be in your own apartment. I'm happy also to be here. I'm imagining, how many times just like have you had to introduce yourself from a speaking point of view? I know you do a lot of talks. Yeah. But, of course, I still always introduce myself. of course. But if you need to count how many times you've done it. A lot. A lot. Exactly. It's like back in Nebonia. Yeah. Cool. But it's always fun. Yeah, definitely. Before, so we're gonna talk about the, as we already mentioned a couple of times, AB testing experimentation, CRO. But to kind of take a starting holistic point, especially with today's world with AI. I'm curious you just mentioned it also You've been here for a while within this field. What your What have you seen from over the last couple of years from an evolution point of view when it comes to AB testing, especially today now that AI has kicked in? Yeah, I think we're living an amazing time since it's so much fun to see AI. And of course, as conversion rate optimization specialists, we love optimising stuff, we love testing stuff, we love experimenting with stuff. So I really like that this industry really embraces AI and uses the new work. I see a lot of benefits. I also see a lot of pitfalls, but yeah, it's a lot of fun to test it, to see where it works best, to make your CRO process more efficient, but also to help others experiment and make experimentation easier and more fun for them. Because you can use AI to speed up yourself, you can also use it to help others run good and proper experiments within organisations. Nice. And could you come with like some concrete examples on where you really see that AI is optimizing or being more efficient as you say, but also where the pitfalls are? Yeah. I think one of the biggest and easiest examples of course before AI development was always our bottleneck. And now for those who do client side AB testing, they can set up their own experiments in no time. And most testing tools already have prompt based experimentation. If they don't have it, you can use a tool like Claude or ChatGPT. So especially for client side experimentation, Developer is not really a bottleneck anymore, but also of course, analyzing big sets of data from user research or reviews or doing your scientific research if you use that for your building your hypothesis, AI is super fast, makes I think a lot more efficient as well as automating stuff in the process. But that's also where the pitfall is because I see groups of people trying to automate everything and let AI do a lot of things with barely any human in the loop really rely on AI too much without checking quality. And there I see a lot of pitfalls where quality really, really drops and you really need to know what quality is in conversion rate optimization to be able to work with AI and know if the output is correct or not. So I still think human in loop is super important and you really need to know quality to see if AI delivers quality. Yeah, makes sense. And have you seen from, so these pitfalls and yeah, making it, more automotive, effective, but and yeah, as you said yourself, there's almost the same road has pros and cons, right? Yeah. Yeah. What I've seen from a marketing perspective is, and I think we've had like also in time, so like three years back or a little bit more than three years back with ChatGPT coming in, oh, now all content can just be produced like, and we can send out 50 blocks per day. That's cool. No, no, we shouldn't do that. We need to have the human perspective in it and so And that kind of had come out also with Claude, especially from the beginning of the year that got a version two revival or what Not a revival, but next step. So like everybody's talking about Claude. And again, so you know, marketing can We can analyze anything for there. We don't need I'm facing sometimes like, oh, why don't you just put everything in your ads And analyze it from cloud, and then it will give you all the highlights and what you need to do and what I also, luckily see with a human eye, not everything is totally correct, and I wouldn't go after that recommendation. I could imagine it's the same from your aspect. Yeah, absolutely. in one prompt you can build a whole team of agents, but it does not mean that those agents are delivering quality. And it's not just only the agent itself, also the data you put into it. Is because again, garbage in is still garbage out. This rule we have had for many, many years and it still applies now and it still applies today. So I can do absolutely wonderful things. It does wonderful things, but yeah, don't get carried away with it and don't believe everything what it says. Do train it so it gets the right quality, but if you know what quality is and you can train it with that to give it the right feedback, then it's fantastic when it starts delivering quality, then it really speeds up the process and it's wonderful to see. I'm super excited about it as well. Yeah. Have you seen also you have your clients talking with them, do you see from a leadership management perspective them being pressured in new ways or you know is there another new expectation? I don't see that yet. Of course, they ask like, do you use AI to work more efficient? And maybe hopefully they also start asking, can you get better results with AI? But with my clients, I don't really see pressure yet. They do get asked for it. I also get asked for companies like, okay, what's your experience with AI in combination with experimentation? So it is important because everyone asks me for a new client as well, but not yet pressure. But I think most people now, as most managers expect people work with AI in the correct way. And yeah, so you don't see like, okay, now we can do a 100 tests experience at once because, we can just push everything in. As I know, I mean myself, it's not doing at least from the same section, doing too many tests at the same time because then you actually don't know what was working or not. Then we get in a very statistical discussion. Believe run as many tests as you And maybe when you run very mature, then start worrying about tests affecting each other, interaction effects. But the funny thing is like I said, development used to be a big bottleneck because we also had to wait for developers to build our tests. Now we can do it ourselves, but I still see companies that run a little bit more tests or maybe even the same amount of tests because they discover there's another bottleneck. For instance, they have a weekly meeting in which they need to get approval for their design and it's still, you go to the meeting of design and make some changes then wait a week again. So the bottleneck just shifted. It could be a bad process. It could be too much stakeholders being involved. So it's really interesting to see where the bottleneck comes now, And for some companies, they manage to really ramp up the number of experiments, but I also see others that only run a few more experiments or the same amount of experiments because there was apparently another bottleneck in the process which they might not have seen before AI. But I've always learned logically with Mouseflow as our product and understanding the behavior on a website, and of course, experiencing is very close to, of course, what we want people to do to understand if the user behavior is better for us and for the client. But I always learned, you shouldn't test too many things at one page because then you wouldn't know exactly. always had that, if you change the button, figure it out. But if you change like 10 different things on the landing page, you wouldn't know exactly If there are 10 different experiments with different hypotheses on different elements, then it could still work, but I don't see a lot of teams already running 10 experiments on one page. But if you put everything in one experiment, yes, that's why you don't know what makes the test get the results as it did, but you can run two or three experiments on the same page as long as you don't have like a blue text color on a blue button in another But different hypothesis and different areas, test it and test as much as you can because the more tests you do, the more winners you will find, the more you will learn, the more successful you'll be. Yeah, okay. So I was not totally off. Not everything in one test, but you can run several types because essentially more tests is more learning is more winners. Yeah. Do you I know I wanted to learn a little bit more. I know that you have written and spoken a lot about the psychological effects. Can you tell us a little bit more on that? I know it's a broad question, but I'm kind of curious on what's kind of like your motivation and passion about it. Oh, there we go. Really back to my start of my CRO career. At that one time I was a marketing student and I was helping a coaching company who coached people about social skills, self esteem, happiness. And I helped them as a marketeer, already started getting interested more in websites because I'm really data driven, result driven, but it also then started wondering like if they can change behaviour of people offline, we should also be able to do it online. But that was already like in 2007, 2008 when I started wondering, okay, so how can we apply psychology on websites to help people make the right decisions? But I was studying marketing and not psychology, so I did think about starting doing second study in psychology, but instead I just started reading all our scientific articles and books about social psychology to learn it myself. And from there, apply it in my work, also write a book about it. And of course, with my former employer, Online Dialogue, had an actual team of psychologists and you could really see the value there because you do change behaviour online in AB tests and behaviour change up psychologists are the best at that and for understanding what happens in the AB test and understanding what users really need. Can you come up with something more concrete within that as examples? of course. So I see a lot of people trying to apply psychology and they just get a social proof or commitment and consistency and they start thinking about experiments, but then there are two fifty psychological biases as they can come up with many, many experiments. It works the other way around. What is really successful is really learning about your users through tool like Mouseflow, through scientific articles, really learning what they need. Then thinking about the underlying psychological principle, which could be consistent, commitment and consistency, but it could also be certainty or motivation or ability. And then grouping experiments, thinking about experiments that relate to that. So it starts with learning about your users through all different research sources, And then thinking like what psychological principle is behind this, what they need, and then can come up with good learnings and follow-up experiments based on that. So if I kind of like looked at my own website, B2B, primarily SaaS business, retail business, e commerce. And I know my ICP, I know the personas, I've done the research, I know that the challenges and the use cases. How do I understand the Psychologic, what I'm gonna say, effect on the data that I'm reading? Yeah. So if you have a psychologist, you have a degree in it, then it's of course much easier and it's much easier to say like, oh, this is this psychological principle, this is this psychological principle. The way I also see a lot with product teams is that they do a lot of research from many different sources. Then they come up with the biggest customer opportunities or customer problems, the most important ones. That's just a sentence like, I'm not sure this product will work for me, or I have no clue what product to choose, or I don't trust the brand. And of course, can name psychological principles which that entails, which psychologists are really good in. But if you just start with that, it start with grouping research and finding the most important opportunities to work on. That already groups experiments together. That's when you get a lot of different ideas than just apply social proof. Or we see this heat map and people don't scroll, so we should do something that makes people scroll. If you group everything together, find those opportunities for biggest customer problems, that's something they really need. So write it from a customer's perspective and then start experimenting with different AB tests that relate to that opportunity on different pages and different steps in the funnel, could even be in marketing and email, that's when you learn what opportunity to address how and where in the customer journey. And can apply many terms, and it could work if you have that knowledge because then you can go a layer deeper, but most teams can already start, should start here. So there will be like, what did you say, it's two fifty? Yeah, like top of my head, I think something like that. Yeah. Yeah, you have a really nice image on I think Wikipedia with all of them grouped. It's a lot. Yeah. Okay. Nice. Is like to me to understand it a bit like, for example, it could be everything from colors that Some colors have another effect on people. So It could be in there, but it could also be things related to first impression, things made to how we make decisions, things made to relate to how we interpret things. So there are a lot. You find a lot of them, of course, in the book Thinking Fast and Slow from Daniel Kahneman that already and Tversky already covers a lot, but of course, are much, much more. Yeah. It's like simple, I guess, that it's not something we think about anymore because it's so much in our culture and in our mindset. I'm just thinking, if you suddenly changed the negative numbers to green and the positive numbers red that would kind of Yeah. Yeah. Yeah. As a very simple example that is actually coming. So it's not something we necessarily think about today, but it is it is based on our psychological reaction, I guess. Yeah, because with every change you make, try to change behavior. And with the behavioral change, there's probably some underlying principle why that behavior change is based on the change you made. Okay, so now we've been a little bit around AI, the psychological effect. How do you see experimentation maturity in the companies that you work with from an early stage and how, is they really are at or where they actually... Yeah. like the question. I just published a report together with Lucius van der Brink where we interviewed 15 industry leaders with lots of experience in experimentation maturity. And we got a lot of interesting findings there. And worked with over 70 brands by now in my career, so I've seen them in every single maturity stage, so to say, but then we will do also finding the report like every situation is different. Like a maturity model is great to start a conversation, but every organization is different. Every organization wants something different, work something different. So I've seen them in every maturity level, but then again, levels are much more complicated than that. Let's just say that there's some people here listening in and they haven't really played that much around experimentation yet. They're still, you sounded wrong. But how should they begin? What is your best advice? And then everything from both, I guess, identifying what experimentation would be the best right now, but also from, I would say, let's say they limited from a tool set point of view. How would you advise them? of all, we must understand that if you just start with an experimentation, you are not responsible for experimentation maturity because that entails a team and department or organization. And of course, when you're just starting, you're probably not the CEO doing some CRO work. external rotation maturity is on many different levels. It's on culture, leadership, on strategy, on scope, on teams, on expertise, tooling, data, everything. So it's really broad, but if you are just starting with experimentations, just starting running those few tests, where your bottlenecks are and start there. Because if you remove those bottlenecks, a more faster, more efficient and more successful, then you have proof to show your manager that what you're doing is really important and that you should maybe get some colleagues who do the same work. And I really believe that change is very useful top down, but it can start with you creating enthusiasm, showing that experimentation really contributes to the business goals, show that experimentation helps your colleagues achieve their goals. So don't push experimentation, tell them to experiment, but show that you can help them achieve their goals with your experiments that shows them value. And when you have more and more people enthusiastic about it, you create a buzz and then your manager, maybe even a higher manager will get more curious and help increase the maturity. And what would you say? So for those that let's say, think they are in a certain level of maturity and think, well, are doing experimentation, it's handled. So if you had, you CMO coming in and say, oh, see where all these zero things come in, are totally on track, it's handled. What would your response be for that? If it's CMO coming in saying, X CRO is handled, I would say you're in the wrong department. Because I think when you're really in experimentation maturity, high in experimentation maturity should be under product or under tech. But with a marketing, rate optimization is still super, important. And if every decision within marketing is made through experimentation and that can be of course, AB testing is one of the experiments, but if you read the book Testing Business Ideas, it covers, I think 45 methods of experimentation and one is AB testing. So it's much broader than that. But if every decision within the marketing department is made based on experiments, everyone can experiment and everyone does experiment and then it's handled. But that's before everyone thinks experiment minded, works experiment minded, and all decisions are made by Xperia, not a lot of companies are there. No, no. I probably, yeah. I could imagine. But it is fun, it's fun. I've been a consultant for many, many years with many, many brands and I really like to help teams and organisations move, more and better decisions with experimentation. it's also with experimentation maturity. Everyone could do an experimentation maturity scan and see where they are approximately on some experimentation maturity model. But understanding where they really are and why they are there and why they're stuck, that is something different. Then also understanding where they need to go. It's not simply even go from level two to level three. Where do they actually need to go? Because every situation is different. That's another step. And then when you're there ensuring it sticks, also when you leave as a consultant or when the champion leaves, that is even harder because then you're really changing behaviour and it's not stuck to one person or one leader. And with the research we did, as I mentioned, we saw that every dimension that they saw experimentation maturity go back, have experienced it, all of them, because a champion left, a leader left. So yeah, scan, everyone can do it, understanding why they're on that level and why they're stuck. This next one, helping them and seeing where they need to go is the next level and then really getting there and making it stick. It's a fun and a challenging puzzle. Yeah. So if you mentioned it a couple of times, you love data, statistics, When do you know that the sample that you've taken and when is an experiment done? When is a single experiment done? Yeah, when do you when can you say, okay, this is actually let's go A and not B. Ah, okay. So we're talking about when is an AB test done? Yeah. Okay, This is something very different, very That's different okay. There are calculators for that where you calculate how long a test should run based on the weekly visitors and weekly conversions. There are many good calculations you can find online. And they calculate the minimum detectable effect. So what's the minimum uplift I can find in my experiments? It's bit statistical. Yeah, that'd go ahead. And you want to have this number as low as possible because if you can detect an uplift of 3%, that's totally fine. If it says 15%, yeah, not very often that you find to uplift And of 15 even in your experiment, you find uplift of 10%, it will still say there's no difference because it can only see 15%. So there's calculators for that to calculate what uplift you should at least find and how many weeks it will take for you. And when you say like in your pre test calculation, have to run for three weeks or for four weeks, then run it for three weeks or four weeks and not longer and not shorter. Otherwise, you're just playing the game and let's wait until it's significant and that's not the way to go. You determine beforehand how many weeks, four weeks, and you stick to that. And because I've experienced from an AB testing point of view that it's like fiftyfifty. What do I do? What do I do if it's fiftyfifty? Fiftyfifty at the end. At the end, yeah. Yeah, then learn from it and see how you can improve. But that's what was the whole idea. What direction should I go? Because it could be that we tried to test, for example, on our pricing pages some things to like, okay, could we nudge them over to just clicks instead? And then we did an AB testing tool and there was kind of no difference. We got as many clicks as usual. Even though we switched, it didn't get the nudging effect that we wanted. What do you then do? Was it just a bad experimentation? No, it's never bad because you tested it. So it's never bad. First of all, would like to see the data, see if you have sufficient data because if you have insufficient data, you And cannot draw the if it is sufficient data and it's still there's no difference, then dig into the data. Maybe your muscle recordings or heat maps or attention maps see a difference or people maybe simply don't care and then come up with better experiments based on your research. Yeah, so for all who are listening in here, what I hear right now is don't just have an AB tool and behavior analytic tools because you might see different data points here and it's not the same answers and analytic basically decisions making that you can actually take based on what you see. Yeah, so just like Mouseflow is critical within the research phase before you run experiments to come with hypothesis, but also during the testing phase, you can see interesting differences with all the data on Mars, but you can even set up a poll for your A and B variant to see how people respond to that. A survey, that's what it's called. Yeah, Yeah, feedback survey. Yeah, Has many names, one of those. we are nearing thirty minutes, I wanted to say. Is there anything you kind of feel like right now going on that I know you could talk for hours. And luckily for us, we have a dinner in a bit. Yeah. So we can discuss even more and talk even more. But, yeah, that you are seeing right now? Well, headed with AI, but besides that. No, I think it's mainly AI and it really also depends on the market where you're at. So where you're listening because a country in Netherlands, we're fortunate that we are quite ahead of the curve, whereas others still have to start with CRO. What I saw over the last years in The Netherlands and seeing more and more other countries is that experimentation and AB testing more and more moving into products, and they're also coming broader with product discovery for bigger innovations, but also AB testing within product itself, which is fantastic because then we're in marketing and in product. So there's more opportunities for more people and that combined with AI, I think, gonna be a good future because also proteins with AI can speed up. If they speed up on the wrong assumption, they'll build a horrible product. So experimentation is becoming more important to first test those assumptions before we start building with AI completely new products. So I think it's a good future and it's fun to combine it with AI and it's fun to test AI within our processes and hopefully within marketing and in products. Nice. And I'm curious, work wise, it's all about experimentation. Is it that in life in general, I know I cheated a little bit, as Claude likes to say, Tell me about Ruben. And it said that you were a football enthusiast. That if I wanted to talk a bit a little bit, that you could probably talk a lot about it. Do you use experimentation? Does it come natural for you to experiment also in your hobbies? Yeah, absolutely. I think it's a mindset. So it's definitely also I use it for sports, for sleep, for food supplements. And it doesn't mean that I'm fully healthy because I also like to have a beer or go to a party. It's all about balance. But yeah, as consultant working a lot, have my girlfriend live in different city and my parents as well want to do sports every single day. It's a good combination, a good experiment to see where I get the most energy, what drains energy, what food supplements I can take, what sport work for me. Do you have a system or a situation in the head? Or do you write down? No, it's mainly in my head. It's not like you are putting all the data and then analyzing afterwards. No, no, no. I do... AI does have access to my calendar, my to do list, and I give feedback about my energy levels, so I do it there. I now have a client in the food supplement industry, so we have a lot of conversations there as well, which is so much fun because they already test with food supplements So it's really easy to explain, oh god, that's what I do for your website as well. Yeah. So it was mainly in my head. And of course, I measure a lot with Fitbit. Fitbit, yeah. So maybe you should do like a month where you are doing this health food agenda program that you're following versus month where you just, well, think we'll know the answer if you just go and drink beer every day and then so on. But... Yeah, the one month of healthy. And then I think you know the I think you know, you've seen my neighborhoods. There's too many bars and restaurants here. That's not gonna happen. Not a necessary experiment. No. Great. Well, enjoyed the conversation. Likewise. It was really nice. And well, you can't... Well, those who are just listening in, you can't see it. But I was so lucky to also get some cookies here. So I will enjoy it after the recording because I think that those who are actually listening in will prefer that I'm not having a cake in my mouth at the same time. But thanks for joining and thanks for the talk. And yeah, look forward to later where we have that dinner with Chameleon, an AB testing tool here in Amsterdam. So, meet a lot of interesting and good people. Yeah, So the conversation is continuing. Exactly. So, Thank you. Thank you. And thanks for listening. Yeah. And if you have questions, feel free to connect with me on LinkedIn. Happy to answer any questions people have. Great. Thank you.