It\u2019s easy to look at the early results of a test and draw a conclusion. Take this example from Peep Laja<\/a> of Conversion XL<\/a>.<\/p>\n What would you do if this was your test? Call the control a winner and start implementing the learnings throughout your site?<\/p>\n That\u2019s what most would do. After all, it says right there that the test version has a 0% chance of beating the control.<\/p>\n A couple of weeks later, the results looked quite different.<\/p>\n That\u2019s right. The version that had no chance of winning was now significantly outperforming the control.<\/p>\n What\u2019s going on here? How can that be?<\/p>\n Scroll back up and look at the sample size again. A little over 100 visitors isn\u2019t enough to get valid results.<\/p>\n So what size do you need?<\/p>\n That depends on a few factors, such as your conversion rate, minimum detectable effect and acceptable margin of error. But the answer is probably higher than you think.<\/p>\n Plugging your numbers into a sample size calculator such as this<\/a> or this one<\/a> is the easiest way to determine the minimum sample size you\u2019ll need.<\/p>\n But this isn\u2019t the only reason to keep your tests running.<\/p>\n Timing is also a variable that could skew results. Some businesses see conversion rates vary significantly between days of the week. There could also be events or other external factors that temporarily skew results one way or the other.<\/p>\n There are also random anomalies.<\/p>\n This simulation<\/a> shows how even A\/A tests (where both variants are identical) can deliver false positives up to 25% of the time.<\/p>\n By letting your tests run for at least a few weeks, any anomalies should even out.<\/p>\n You\u2019d think this would be the last thing you\u2019d see on a list of testing mistakes. After all, the whole reason for testing is to avoid assumptions and act on facts.<\/p>\n But even the experts sometimes fall into this trap.<\/p>\n Take this example from the Unbounce blog<\/a>.<\/p>\n The article makes it sound like it\u2019s testing the use of video. But that\u2019s not the only difference between the versions.<\/p>\n My hypothesis would be that the headline in the winning version had a bigger impact. That\u2019s the first thing that jumps out at me. (Although I\u2019m a copywriter so words are kind of my thing.)<\/p>\n If you want to learn definitively how different components affect response, you need to isolate them.<\/p>\n Don\u2019t get me wrong. There\u2019s nothing wrong with testing drastically different creatives against each other. In many cases, it\u2019s a better choice than testing a single component, because it offers a greater chance of a significant lift.<\/p>\n Just be aware what the test is: The sum of the differences, not each individual variation.<\/p>\n We\u2019ve all seen articles that highlight results like this:<\/p>\n It\u2019s pretty exciting. So it\u2019s understandable to want to rush out and test it on your own site.<\/p>\n Here\u2019s why you shouldn\u2019t.<\/p>\n First, trying to replicate a test you read about is a spitball approach. There\u2019s no strategy behind it. You\u2019re just making changes to see what sticks.<\/p>\n Smart testing starts with a valid hypothesis. And no, \u201cthis worked for another company so it might work for us\u201d is not a valid hypothesis.<\/p>\n A hypothesis should start with some sort of insight or logic. For example, if customer interviews reveal a common pain point or purchase driver, you might hypothesize a headline that speaks to that would perform better than the control headline.<\/p>\n Even smaller tests should include an educated rationale, such as \u201cusing a color with more contrast for the CTA button should help draw more attention to what we want visitors to do.\u201d<\/p>\n Second, and even more importantly, small changes usually only lead to small wins. To get big wins, you need to test the big things first. For most of us, changing the color of a button just isn\u2019t going to make a huge difference. Wasting your time on that deprives you of other important learning opportunities.<\/p>\n Remember, accurate testing requires a large sample size. And there\u2019s only so much traffic to go around.<\/p>\n A framework such as PIE<\/a> or PXL<\/a> can help you prioritize your hypotheses and ensure you\u2019re getting the most out of your testing.<\/p>\n It\u2019s easy to get tunnel vision when you\u2019re looking at A\/B test results. After all, you\u2019re supposed to isolate variations so you know exactly how a change impacts your conversion rate.<\/p>\n The problem is not in the testing itself, though. It\u2019s in how you interpret and implement those results.<\/p>\n Let\u2019s use a hypothetical example.<\/p>\n Say you\u2019re testing two different offers for your landing page; a free trial and a free demo.<\/p>\n Version A, the trial, gets a response of 3.2%. Version B, the demo, gets 5.6%.<\/p>\n Based on those results Version B is the clear winner. Would you say that Version B is the better offer for your business? Would you roll it out across your site?<\/p>\n Maybe the demo is better. But maybe it\u2019s not?<\/p>\n What if the leads from Version B mostly fail to purchase afterward, while a good portion of Version A\u2019s become paying customers?<\/p>\n Or what if the leads that do become customers from the demo churn at a higher rate than the trial?<\/p>\n Would you still say Version B is the better offer?<\/p>\n Here\u2019s another example.<\/p>\n Let\u2019s say you have an Adwords campaign running, and your headline is \u201cStart Testing in Minutes.\u201d<\/p>\n Now let\u2019s say that ad leads to a landing page where you\u2019re running an A\/B test on the headline.<\/p>\n Version A is: \u201cStart Testing in Minutes with This Point-and-Click A\/B Testing Tool\u201d<\/p>\n Version B is: \u201cIncrease Conversions with This Point-and-Click A\/B Testing Tool\u201d<\/p>\n Version A wins pretty handily. So you should use that approach on all your pages, right?<\/p>\n Not necessarily. Sure, A is the winner here. But you need to consider the factors beyond the page. Look at the headline of the ad that drove people to that page again.<\/p>\n Version A is the better match. It gives the people the information they expect, moving them seamlessly through the journey.<\/p>\n Version B on the other hand is a change in direction. It breaks the flow, killing momentum and conversions.<\/p>\n But what if the Adwords headline matched Version B closer? The results would almost assuredly be different.<\/p>\n After declaring a winner, what do you do? I bet you roll out the winning version and move on to something else. But then you\u2019re missing out on valuable insights.<\/p>\n One of the first things you should do is share the results with the rest of your team. I can\u2019t tell you how many times in my own career some useful nugget has come to light after a project was finished or well under way.<\/p>\n You also want to apply what you learn elsewhere. Try to replicate the results with other segments. Use it to form new hypotheses. Build on those tests by further refining your most important pages. Optimization is an iterative process.<\/p>\n Finally, make it a point to revisit your tests periodically. Things change rapidly in today\u2019s world and what\u2019s working now might not be the best option in a year or two.<\/p>\n If you\u2019re making any of the mistakes I\u2019ve just described, don\u2019t feel bad. Most marketers make at least one, even the ones from successful companies.<\/p>\n The good news is that now you know what you were doing wrong and you can correct course. Here\u2019s the first steps you can take today:<\/p>\n Let me know how it goes in the comments section below.<\/p>\n A\/B testing is sort of like an American Idol audition. Everyone thinks they\u2019re great at it. Few actually are. I\u2019m sorry to […]<\/p>\n","protected":false},"author":1,"featured_media":476,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30],"tags":[],"yoast_head":"\n![\"Early](\"https:\/\/mlzf3ramde6k.i.optimole.com\/w:auto\/h:auto\/q:mauto\/ig:avif\/https:\/\/i0.wp.com\/www.seankirbycopy.com\/wp-content\/uploads\/2016\/11\/EarlyConversion.jpg?resize=620%2C217&ssl=1\")
<\/a><\/p>\nImage source: Conversion XL<\/a><\/pre>\n
![\"Test](\"https:\/\/mlzf3ramde6k.i.optimole.com\/w:auto\/h:auto\/q:mauto\/ig:avif\/https:\/\/i0.wp.com\/www.seankirbycopy.com\/wp-content\/uploads\/2016\/11\/LaterConversion.jpg?resize=690%2C216&ssl=1\")
<\/a><\/p>\nImage source: Conversion XL<\/a><\/pre>\n
Mistake #2: You\u2019re making assumptions<\/h3>\n
![\"Aussie](\"https:\/\/mlzf3ramde6k.i.optimole.com\/w:auto\/h:auto\/q:mauto\/ig:avif\/https:\/\/i0.wp.com\/www.seankirbycopy.com\/wp-content\/uploads\/2016\/11\/Aussie-A-B-Test-Results.png?resize=799%2C640&ssl=1\")
<\/a><\/p>\nMistake #3: You\u2019re testing the wrong things<\/h3>\n
![\"A\/B](\"https:\/\/mlzf3ramde6k.i.optimole.com\/w:auto\/h:auto\/q:mauto\/ig:avif\/https:\/\/i0.wp.com\/www.seankirbycopy.com\/wp-content\/uploads\/2016\/11\/Button-Test.png?resize=788%2C444&ssl=1\")
<\/a><\/p>\nImage source: Leadpages<\/a><\/pre>\n
Mistake #4: You\u2019re missing the big picture<\/h3>\n
Mistake #5: You treat testing as a one-step process<\/h3>\n
Conclusion<\/h3>\n
\n
Feature image by Alex Proimos<\/a><\/pre>\n","protected":false},"excerpt":{"rendered":"