Mouseflow Podcast: Uncovering Website Analytics
"Mouseflow Podcast: Uncovering Website Analytics," a podcast by Mouseflow, provides actionable strategies to optimize website performance and enhance user experience. Covering conversion tracking, user behavior analysis, A/B testing, data visualization, and more, it offers valuable insights to improve online presence and drive meaningful results.
Mouseflow Podcast: Uncovering Website Analytics
AI in CRO What Experts Really Think (and How It’s Actually Being Used)
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
In this episode, we explore the reality of AI in CRO and debunk the myth that AI will replace CRO professionals. Tune in to learn why human judgment still wins in strategic decision-making and how the future involves Humans + AI working smarter.
Will AI Replace CRO?
SpeakerWelcome back to the deep dive. Okay, I want to get straight into it today. Let's do it. Because the topic we're covering is, well, it's pretty much the only thing anyone in our industry is talking about right now.
Speaker 1It really is. It feels like the ground is shifting under our feet.
SpeakerYeah, and I don't mean that in a, hey, check out this cool new tool kind of way. I mean it in a uh, is my career going to exist in five years kind of way.
Speaker 1That's the real question, isn't it? The anxiety is palpable.
SpeakerAbsolutely. We are looking at artificial intelligence, but you know, specifically how it's crashing into the world of conversion rate optimization. CRO.
Speaker 1Yeah.
SpeakerAnd I have to be honest, when I first picked up this report, it's called AI and CRO, what experts really think. I was bracing myself for some bad news.
Speaker 2I can imagine.
SpeakerBecause if you just look at CRO on paper, what is it? It's data analysis, it's A/B testing, pattern recognition. I mean, that sounds exactly like a job description for a super powerful algorithm.
Speaker 1It seems like the perfect candidate for automation, at least on the surface. And you're right, the report opens right up with that sentiment. It says something AI will replace CRO.
SpeakerRight.
Speaker 1It's not a fringe theory, it's like the dominant anxiety.
SpeakerSo let's not beat around the bush. Is this a eulogy for the human optimizer? I mean, I see tools now that claim to auto-generate, copy, auto-design variants, auto-allocate traffic. What is left for us to do?
Speaker 1Well, maybe put the resignation letter away for a minute. Okay. Because once you actually talk to the experts, the people deploying these tools at a huge scale, the whole narrative, it just changes completely.
SpeakerOh so?
Speaker 1The consensus in this deep dive is surprisingly very firm. AI isn't taking over optimization.
SpeakerOkay. I have to push back on that immediately. That sounds a little like wishful thinking.
Speaker 1I get.
SpeakerWe're talking about machines that can process petabytes of data in seconds.
Speaker 1I can't do that. You can't do that. Why aren't
AI Can’t Explain the “Why”
Speaker 1we just admitting the machine is better at the math?
SpeakerBecause CRO isn't math. That's the entire misconception.
Speaker 1If CRO were just make the number go up, then yes, the AI would win hands down. But the experts here, they argue that CRO is actually about understanding real people.
SpeakerUnderstanding real people.
Speaker 1And that is a very, very different data set.
SpeakerIs it though? I mean, everything we do online, it leaves a data trail. Clicks, scrolls, heat maps. Isn't understanding people just a fancy way of saying aggregating enough behavioral data?
Speaker 1Ah, but that's the trap. That's thinking of people as just, you know, wallets with cursors attached.
SpeakerRight.
Speaker 1The report makes a crucial distinction here. AI is phenomenal at telling you what happened. It can look at your dashboard and say conversions dropped 15% on mobile between 2 and 4 p.m.
SpeakerWhich is useful.
Speaker 1Incredibly useful, but it cannot tell you why. And the why usually lives completely outside the data set.
SpeakerGive me a concrete example because I feel like AI is getting better at the why every single day. You know, sentiment analysis and all that.
Speaker 1Okay, sure. Let's imagine a B2B sauce company, high-value enterprise pricing page. All of a sudden, conversion rates just tank the data, the AI's world. It shows that time on page is the same, scroll depth is the same, but nobody is clicking book a demo.
SpeakerSo the AI would suggest what? Changing the button color, shortening the headline.
Speaker 1Exactly. It starts iterating on the page elements. It tries to optimize the local funnel, but the human optimizer, who understands context, realizes something else. Which is that the company just announced a shift in their service level agreement that kind of screws over small businesses. And the whole market is reacting to a press release that isn't even on your website.
SpeakerWow. Okay, so the AI is optimizing a burning building.
Speaker 1Precisely. The source material calls this the context blind spot. AI can't fully understand user context because user context is the entire human experience. It's not just the session data.
SpeakerSo it hits this hard ceiling, it can optimize locally on the page, but it fails globally because it just doesn't know what's happening in the real world.
Speaker 1It lacks cultural and emotional peripheral vision. It doesn't know it's a holiday, or that a competitor just launched a free version of your product.
SpeakerOr that the phrase you're using in your headline just became a political slur on Twitter yesterday.
Speaker 1Yes, exactly. That's a perfect example.
SpeakerOkay, that makes sense. But let's play devil's advocate for a second. We know LLMs are getting smarter, they have access to the news, social sentiment. Isn't this blind spot just a temporary glitch? Won't like GBT six or seven eventually connect those dots?
Speaker 1It might connect the data points, but can it weigh them? That's the real issue. It's about discerning signal from noise. If you feed an AI everything news, weather, stock prices, it starts finding these correlations that aren't causal at all. It might tell you conversions dropped because it's raining in Seattle.
SpeakerAnd a human knows. That's just ridiculous.
Speaker 1A human knows it's irrelevant. The report really emphasizes that context is king. And for right now and for the foreseeable future, we're the only ones who can carry that crown. We understand motivation. AI just understands action.
SpeakerI like that distinction, motivation versus action. But this this brings us to a weird paradox that the report mentions. If AI is this powerful tool, but it's kind of dumb about the why, there's a real danger there.
Speaker 1A huge
The Danger Zone of AI
Speaker 1danger.
SpeakerThey call it blindly trusting AI answers.
Speaker 1This is the danger zone. And honestly, this is where I see a lot of smart professionals getting burned.
SpeakerBut isn't the whole sales pitch of these tools that we can trust them? You know, let our AI algorithm optimize your ad spend. Let our AI write your winning copy.
Speaker 1Of course.
SpeakerWe buy the tools specifically so we can be passive. Are you saying that whole sales pitch is a lie?
Speaker 1I'm saying the sales pitch is dangerous if you swallow it whole. The report warns against accepting the output without asking the right questions. The failure mode here is just passivity.
SpeakerIt's the set it and forget it mentality.
Speaker 1Right. And think about the speed we all operate at. You're under pressure to launch a campaign. You ask the AI for five headline variations, it spits them out, they look fine.
SpeakerThey look fine.
Speaker 1Grammatically correct, they contain the keywords, so you ship them.
SpeakerAnd I just saved myself two hours of brainstorming.
Speaker 1You did. But you stopped thinking. You didn't ask, is this tone actually aligned with our brand voice, or does it just sound like a generic sales bot? You didn't ask, is this making a promise our product can't actually keep? Right. The AI doesn't have a conscience, and it certainly doesn't care about your brand equity. It just predicts the next likely word.
SpeakerSo the real danger isn't that the AI goes rogue and I don't know, destroys the website. The danger is that we get lazy and let mediocrity slide because it was generated so efficiently.
Speaker 1Efficient mediocrity. That is a great way to put it. The experts in the report, they stress that using AI actually requires more critical thinking from us, not less.
SpeakerWhich is so ironic, right? I want the tool to reduce my cognitive load, but you're telling me I need to ramp up my skepticism.
Speaker 1Your role just shifts. You stop being the generator and you start being the editor.
SpeakerAnd being a good editor is hard work.
Speaker 1It's very hard work. You have to verify, you have to challenge, and you have to have the confidence to say, no, this is wrong, even when the machine is telling you it's statistically probable.
AI’s Sweet Spot
SpeakerOkay, so we've established that AI is basically a sociopath with no context, and we shouldn't trust it blindly. Which sounds pretty bleak.
Speaker 1It does a bit, yeah.
SpeakerBut the report isn't anti-AI, is it?
Speaker 1Not at all. In fact, once you get past all the warnings, the experts are incredibly bullish on it. But they are bullish on very specific use cases. Okay. The report outlines what it calls a sweet spot, three key areas where AI doesn't just help, it absolutely dominates.
SpeakerLet's break these down because this is the practical stuff people need. The first one is velocity.
Speaker 1Or move faster.
SpeakerOkay, but I want to be careful here. We just talked about how speed can be dangerous shipping bad headlines fast. So how is velocity a benefit?
Speaker 1Think about the blank page problem. How much time do you personally spend staring at a blinking cursor, just trying to come up with the first hypothesis for a test?
SpeakerWay too much. And by the time I come up with like three ideas, I'm already exhausted.
Speaker 1Exactly. AI removes the friction of initiation. It can generate 50 hypotheses in 10 seconds.
Speaker50.
Speaker 150. Now, 40 of them might be total trash, but 10 of them might be gold. A human might never have even reached those 10 gold ideas because they burned out on idea number four.
SpeakerSo it's about increasing the volume of experimentation.
Speaker 1It's about accelerating the build phase of that build, measure, learn loop. If you can test 10 times as many variations, you just statistically increase your chances of finding a winner.
SpeakerProvided you are filtering them with human judgment.
Speaker 1That's the key. Always. And there's a technical side of velocity too, right? Like multivariate testing.
SpeakerOh, for sure. Trying to test four headlines against three hero images and two button colors.
Speaker 1Doing that setup manually is a total nightmare. AI just automates the permuting. It lets you run these complex experiments that, you know, just wouldn't be feasible for a human team to manage on their own.
SpeakerOkay, so velocity is about removing the bottleneck of creation and setup. What about the second area? Vision.
Speaker 1This one is my favorite. Spot patterns. This is where AI just leaves humans in the dust.
SpeakerHow so? I mean, we're pretty good at spotting patterns. I could look at a graph and see a downward trend.
Speaker 1Sure, you can see a trend in two dimensions, maybe three. You can see traffic versus time or conversion versus device, but can you see the correlation between time a day, referral source, browser version, and cart value all at the same time?
SpeakerMy brain hurts just trying to visualize that chart.
Speaker 1Exactly. We have cognitive limits. AI operates in like n-dimensional space. It can look at a data set and say, hey, do you notice that users on iOS 17 coming from LinkedIn are bouncing at a 90% rate? But only if they land on the pricing page.
SpeakerThat is a needle in a haystack.
Speaker 1It's a needle in a stack of other needles. A human analyst might spend a week pivoting tables in Excel to find that. Or they might never find it because they didn't even think to look for that specific combination.
SpeakerBut the AI doesn't need to think to look.
Speaker 1It just processes everything and flags the anomalies.
SpeakerSo in this case, the AI is the detective, and it's finding clues that are literally invisible to the naked eye.
Speaker 1It creates vision where we are blind. It surfaces the questions that we should be asking.
SpeakerOkay, philosophies. The third one is synthesis, to summarize insights.
Speaker 1This is all about the unstructured data problem.
SpeakerUnstructured data. You mean like customer reviews and things like that?
Speaker 1Reviews, support tickets, chat logs, open-ended survey responses, all the qualitative stuff.
SpeakerRight.
Speaker 1In the old days, and by old days I mean like two years ago, if you wanted to analyze qualitative data, you had to read it. All of it.
SpeakerAnd let's be real, nobody reads all of it. You read the first 50 reviews, you get bored, and you assume you know what everyone is saying.
Speaker 1Or you just cherry pick the reviews that confirm what you already believe. Look, this guy loves the new feature while ignoring the 200 people who hate it.
SpeakerGuilty as charged.
Speaker 1AI just changes the physics of this. You can feed it 5,000 support tickets from the last month and just ask, what are the top three friction points for users trying to upgrade?
SpeakerAnd it doesn't get bored.
Speaker 1It doesn't get bored. And it has no confirmation bias. It just gives you a synthesized view. Forty percent of users are confused by the tier naming convention. Now that is an actionable insight that might have been buried forever.
SpeakerSo synthesis is really about listening at scale.
Speaker 1That's it. It turns all that noise into a clear signal.
SpeakerYou know, I'm looking at these three benefits velocity, vision, and synthesis. And what strikes me is that none of them are strategy. Velocity is execution, vision is observation, synthesis is organization, but not one of them is actual
Why Human Judgment Still Wins
Speakerdecision making.
Speaker 1And that brings us to the verdict of the entire report. If there is one sentence I want people to like tattoo on their forearms, it's this one. It can't make decisions for you.
SpeakerYeah, the report is very firm on this. Human judgment still wins.
Speaker 1Let's really unpack that word judgment, because I think people sometimes confuse judgment with prediction.
SpeakerRight. If I ask ChatGPT which of these two headlines is better, and it picks one, isn't that a judgment?
Speaker 1No. That is a probabilistic prediction based on its training data. It's just guessing which one is statistically more likely to align with patterns of good headlines it has seen in the past.
SpeakerSo what is judgment then?
Speaker 1Judgment is weighing consequences. Judgment is ethics, judgment is strategy. Let's go back to that Saw's pricing page example. The AI might predict if you make it impossible to cancel the subscription online and force people to call a phone number, your retention rates will go up.
SpeakerAnd statistically, that's probably true. Dark patterns work.
Speaker 1Exactly. The math checks out. But a human using judgment says, yes, retention will go up for three months, but our brand reputation will be destroyed, our trust pilot score will tank, and we'll probably get sued.
SpeakerSo the AI optimizes for the metric, but the human optimizes for the business.
Speaker 1That's it perfectly. The AI creates the map. It says this is the fastest route. The human driver says, yeah, but that route goes through a swamp and I'm driving a sedan. That is judgment.
SpeakerIt's funny because for years we've been told that to survive, we need to be more data-driven, more like computers. But this report is suggesting the complete opposite. It's saying to survive the AI era, we need to be more human.
Speaker 1It's the ultimate pivot. The value of the person listening to this, your value, isn't in your ability to build a pivot table anymore. The machine can do that. Your value is in your ability to look at that table and ask, so what?
SpeakerThe so what question.
Speaker 1That's the money question. The AI says mobile traffic is up. You ask, so what? The AI says bounce rate is down. You ask, so what? Does that mean they're engaging, or does that mean they just can't find the logout button? Meaning is the exclusive domain of the human.
SpeakerThis reframes the entire career path, doesn't it? I feel like a lot of people in CRO and marketing, they've built their identities around being the technical person.
Speaker 1For sure, the one who knows how to set up the tags and run the scripts.
SpeakerYeah.
Speaker 1And look, those technical skills are still useful, but they are becoming commodities. The skills that are gaining value are the so-called soft skills, empathy, curiosity, strategic thinking, the ability to argue with the data.
SpeakerArguing with the data, I love that.
Speaker 1You have to. You have to be the skeptic in the room. The future isn't AI versus humans. The report sums it up perfectly. It's humans plus AI working smarter.
SpeakerIt's a cyborg model.
Speaker 1It really is. You let the AI handle all the grunt work, the pattern spotting, the initial drafting, the massive data crunching, and that frees you up to do the job you were actually hired to do in the first place.
SpeakerWhich is to understand the customer and grow the business.
Speaker 1Exactly. Nobody got into marketing because they loved manually tagging 5,000 support tickets. God knows. So let the robot do the robot work. You do the human work.
SpeakerIt sounds like we're all getting a promotion. We're being promoted from data processor to head of strategy.
Speaker 1Or head of judgment. I think that's the title we should all be aiming for.
SpeakerHead of judgment. I like the sound of that. It commands a certain salary, I would think.
Speaker 1It certainly should.
SpeakerSo we've gone from this fear of being replaced to realizing we're actually being augmented. We know AI gives us velocity, vision, and synthesis. But we also know it has this massive blind spot with context, and it can be dangerous if we trust it passively.
Speaker 1Right. And ultimately, the guardrael is you, the human in the loop.
SpeakerOkay, before we wrap up, I want to leave everyone with a bit of a challenge here.
Speaker 1Ooh, I like it.
SpeakerWe've been talking about how AI takes away the grunt work so we could focus on the high-level thinking.
Speaker 2Right.
SpeakerBut that's actually a really scary proposition for some people. Because the grunt work is safe, it's busy work. You can hide behind a spreadsheet for eight hours and feel productive.
Speaker 1That is very true.
SpeakerSo the question to you is if the AI takes away all the busy work, are you ready to step up? When you strip away the tasks that the machine can do, is your empathy, is your strategy, is your judgment actually sharp enough to carry the full load?
Speaker 2That is the provocative question. There is nowhere to hide anymore. If you're not bringing insight, you're not bringing value.
SpeakerThe machine is doing its part. Now it's time for us to do ours. Think about that next time you open your dashboard. Thanks for listening to the deep dive. We'll see you next time.
Speaker 2Stay curious.