You redesign your enquiry page, enquiries go up the next month, and everyone agrees the new design works. Then you notice the school holidays ended that month and a competitor paused their Google Ads. Was it the design? You can’t tell.
A/B testing answers that question properly. This guide explains how it works and how to run one on a typical Singapore SME website, with most of the space given to the part other guides skip: what to do when your site gets a few thousand visitors a month, not a few hundred thousand.
If you’re new to conversion work in general, start with our explainer on what conversion rate optimisation is. This article zooms in on the testing step.
The short answer
- A/B testing shows two versions of a page (or ad, or email) to randomly split groups of visitors at the same time, then compares which version gets more of the action you care about.
- Version A is the control (what you have now). Version B is the variant with one change. Because both run in the same weeks, outside factors like holidays or ad changes hit both equally.
- You need enough visitors and conversions before the result means anything. A page with 2,400 visitors a month and a 4% conversion rate needs around six months to reliably detect a 25% improvement.
- Low-traffic sites can still test. Test bigger changes, measure a more frequent action (such as WhatsApp button clicks), or run the test inside Google Ads or Meta Ads where you control the traffic.
- Google Optimize is gone. Google shut it down on 30 September 2023. In 2026 the realistic options are paid tools like VWO, Optimizely, Convert and AB Tasty, built-in features such as Shopify Rollouts, WordPress plugins, and the experiment tools inside Google Ads and Meta Ads Manager.
What is A/B testing?
A/B testing (also called split testing) is a controlled experiment. You create a second version of a page with one change, and a testing tool sends each visitor to one version at random (and keeps showing them the same one). After enough visitors, you compare the conversion rates.
Three ingredients make it a real test rather than a guess:
- Random assignment. Visitors don’t choose the version, and you don’t send “good” traffic to one side.
- Running both versions at the same time. A before-and-after comparison mixes your change with everything else that changed that month.
- A single goal decided in advance. For example, “submitted the quote form” or “completed checkout”, not “whatever metric looks best at the end”.
For many Singapore service businesses the conversion is a form, a WhatsApp click or a tap-to-call, not a checkout. Track it before you test anything. Our guides on setting up Google Analytics 4 and tracking website forms and WhatsApp enquiries cover the setup.
A/B vs split URL vs multivariate testing
| Method | What changes | How it works | Traffic needed | Best for |
|---|---|---|---|---|
| A/B test | One element or a small group of related elements on the same page | The tool swaps content on the same URL for half the visitors | Moderate | Headlines, CTA wording, form length, hero image |
| Split URL test | A whole page | Visitors are sent to two different URLs (for example /quote/ and /quote-new/) | Moderate | A full redesign of a landing page or a new page template |
| Multivariate test (MVT) | Several elements at once, in every combination | Two headlines × two images × two buttons = eight versions | Very high | Large ecommerce or SaaS sites fine-tuning a page that already works |
For most SMEs, plain A/B and split URL tests are the practical options. Eight MVT combinations need roughly four times the traffic of a two-version test, so skip MVT if your page gets a few thousand visitors a month. (An A/B/n test, with three or more versions, has the same problem on a smaller scale.)
A note on SEO. Google’s guidance on website testing says not to cloak (show Googlebot something different from users), to put a rel=“canonical” link on variant URLs pointing to the original, to use 302 (temporary) redirects rather than 301s for split URL tests, and to run tests only as long as needed. Follow those four rules and testing won’t hurt your rankings.
What to test first on an SME website
Don’t test button colours on a page with a bigger problem. Start with the pages most visitors see just before they convert.
Step 1: find your test page. In GA4, open Reports > Engagement > Landing page (or Pages and screens) and look for pages with high traffic and a key event (conversion) attached. For a service business that’s usually the homepage, a main service page and the quote page. For a shop, top product pages and the cart.
Step 2: look for the leak. Watch 20 to 30 session recordings in Microsoft Clarity (free) or Hotjar on that page. Note where people stop scrolling, what they tap that isn’t a link, and where they abandon a form. Our article on web design psychology and user behaviour explains what these patterns usually mean.
Step 3: pick from this list, roughly in order of how often each one moves the number on SME sites:
| Priority | What to test | Example for a Singapore SME | Why it often matters |
|---|---|---|---|
| 1 | The offer and headline above the fold | “Aircon chemical wash, same-week slots in the East” vs “Singapore’s trusted aircon specialists” | Visitors decide in seconds whether they’re in the right place |
| 2 | The main call to action | “WhatsApp us for a quote” vs “Request a quote” form | Many Singapore customers prefer chat to forms |
| 3 | Form length and fields | Five fields vs three (name, mobile, service) | Every extra field, especially NRIC or full address, loses people |
| 4 | Proof near the CTA | Google review rating and count beside the button vs at the bottom of the page | Trust signals work best where the decision happens |
| 5 | Price information | “From $X per unit” vs “Contact us for pricing” | Hiding all pricing can send price-checkers to competitors |
| 6 | Page length and order | Short page vs long page with FAQ and process steps | High-value services often need more explanation |
| 7 | Mobile layout | Sticky bottom bar with call and WhatsApp buttons vs none | On most SME sites we see, phones bring the bulk of traffic. Check your GA4 device report |
The “$X” is a placeholder: test the format, using your own prices.
Fix poor page speed before testing, since a testing script makes a slow page slower (see our website speed optimisation guide). And some problems, like a broken form, don’t need a test at all. Run through the CRO checklist and just fix those.
How to write a test hypothesis
A hypothesis forces you to say what you’re changing, why, and how you’ll judge it. Without one, you end up running random tests and reading meaning into noise.
Use this template:
Because we saw [evidence], we believe changing [element] to [new version] for [audience] will [increase/decrease] [primary metric], because [reason]. We’ll know it worked if [metric] changes by at least [minimum effect] after [sample size].
A worked example for a home renovation company:
Because Clarity recordings show 60% of mobile visitors scroll past the quote form without tapping it, and our WhatsApp chats show most people ask about price first, we believe replacing the five-field form with a “Get a WhatsApp quote” button and a “Typical HDB kitchen: from $X” line for mobile visitors will increase quote requests, because it answers the price question and removes typing. We’ll know it worked if mobile quote requests (form plus WhatsApp clicks) rise by at least 30% once each version reaches the sample size we calculated.
Checklist for a good hypothesis:
- Based on evidence (analytics, recordings, customer questions), not taste.
- One primary metric decides the winner. Watch others, but don’t switch.
- A change a customer would actually notice.
- Minimum effect and sample size set before launch, so nobody moves the goalposts.
Sample size and test length
This is where most SME tests go wrong. Four numbers decide how many visitors you need:
| Input | What it means | Typical setting |
|---|---|---|
| Baseline conversion rate | Your current rate on the page being tested | From GA4, last 4 to 8 weeks |
| Minimum detectable effect (MDE) | The smallest relative improvement you want to be able to detect | 20% to 50% for small sites |
| Significance level | How much risk of a false winner you accept | 5% (often shown as “95% confidence”) |
| Statistical power | The chance of detecting a real effect of your MDE size | 80% |
A free calculator such as Evan Miller’s sample size calculator, or the one inside your testing tool, does the maths. But it helps to see the numbers for a realistic small site.
Worked example: a tuition centre trial-class page
A tuition centre in Bishan gets 2,400 visitors a month (about 550 a week) to its free trial class page. 4% of visitors book a trial, so about 96 bookings a month. The owner wants to test a new headline and a shorter booking form.
Using the standard two-proportion formula at 95% confidence and 80% power:
| If the new version lifts bookings by… | From 4% to… | Visitors needed per version | Total visitors | Test length at 550/week |
|---|---|---|---|---|
| 25% | 5% | about 6,700 | about 13,500 | about 24 weeks |
| 50% | 6% | about 1,900 | about 3,700 | about 7 weeks |
| 100% | 8% | about 550 | about 1,100 | about 2 weeks (run at least 2 full weeks anyway) |
So if the new page is only a little better (96 to 120 bookings a month), this site needs about six months to prove it, and exam seasons and school holidays will muddy the result long before then. If the change is bold enough to lift bookings by 50%, seven weeks is realistic. The lesson for small sites: test big changes, not small tweaks.
We’ve used a lead-generation example with different numbers in our guide to landing page CRO for Singapore lead-gen campaigns, which also includes an ICE scoring method for prioritising tests.
Rules for test length
- Run whole weeks, at least two, even if you hit the sample size sooner. Weekday and weekend visitors behave differently.
- Avoid unusual periods unless that’s what you’re testing: Chinese New Year, the Great Singapore Sale, 11.11, the June and December school holidays, or the week a big ad campaign starts.
- Set an end date before you start and stick to it.
- Stop after about eight weeks if you haven’t reached the sample size, because cookie deletion and device switching blur the groups. Call it “no clear difference” and try a bolder test.
What to do when your traffic is too low
If the calculator says you need a year, don’t run the test and hope. Try these instead.
- Test a bigger change. Rewrite the whole offer, change the page structure, or switch from form to WhatsApp. Larger effects need far fewer visitors.
- Measure a more frequent action. If 12% of visitors click “Book a trial” but only 4% finish booking, test against the click. At a 12% baseline, detecting a 25% lift needs about 4,100 visitors in total instead of 13,500. The trade-off: more clicks don’t always mean more bookings, so check final bookings too.
- Test across similar pages at once. If you have ten service pages with the same layout, apply the variant to all ten as one test. Template-level tests pool the traffic.
- Test inside your ads. Google Ads and Meta Ads let you split paid traffic and test landing pages or messages. You control the volume through budget, so you can reach a sample size faster. Use the winning message on the website afterwards.
- Do qualitative research instead. Watch five people who match your customers try to book on your site (in person or on a video call). You’ll find the worst problems in an afternoon.
- Make the change and monitor it. For changes backed by strong evidence, launch and compare the next eight weeks with the previous eight and with last year. Label it a before-and-after comparison, not a test.
Statistical significance in plain words
When a tool says a result is “statistically significant at 95%”, it means: if the two versions actually performed the same, a difference this large would show up by chance less than 5% of the time. It’s a guard against being fooled by luck.
What it does not mean:
- It doesn’t mean there is a 95% chance B is better.
- It doesn’t tell you how much better. A “significant” 30% lift often has a range like “somewhere between 5% and 55%”. Look at that range (the confidence interval), not just the headline number.
- It doesn’t hold if you checked daily and stopped the first day it turned green.
Some tools report Bayesian results instead, such as “probability to beat baseline: 96%”. That reads more naturally, but the same rule applies: set the sample size first and don’t stop early.
Sanity check: with a 50/50 split, visitor counts should be close to 50/50. If one version got 5,400 and the other 4,600, something is broken (a failing redirect, caching, bot filtering). Tools call this a “sample ratio mismatch”. Fix it and restart.
A/B testing tools in 2026
Google Optimize, the free tool many SMEs used, shut down on 30 September 2023 and Google didn’t replace it with a free equivalent. GA4 measures results but can’t run website experiments by itself. Here’s what’s available now.
| Tool | Type | Good fit for | Things to know |
|---|---|---|---|
| VWO | Testing platform with visual editor, heatmaps and recordings | SMEs and mid-size sites wanting an all-in-one tool | Priced by monthly tracked users. Check current plans on VWO’s site |
| Optimizely | Enterprise experimentation platform | Large sites and teams with developers | Quote-based pricing aimed at larger companies |
| Convert | Testing platform focused on privacy and speed | Agencies and mid-size sites, ecommerce | Paid plans with a free trial |
| AB Tasty | Testing and personalisation platform | Mid-size and large ecommerce brands | Quote-based pricing |
| Shopify Rollouts | Built into Shopify admin (Markets > Rollouts) | Shopify stores testing a new theme or checkout setup | Basic plan and above, per Shopify’s changelog. Covers checkout setups too since June 2026 |
| Nelio A/B Testing | WordPress plugin | WordPress sites testing pages, headlines and themes | Free tier: 500 tested page views a month, enough to try it. Paid plans for real tests |
| Google Ads experiments | Built into Google Ads | Testing landing pages, bidding or ad changes on paid search traffic | Splits a campaign’s traffic and budget. Search, Display, Video and Hotel campaigns, one experiment at a time |
| Meta A/B test | Built into Meta Ads Manager | Testing creative, audience or placement for Facebook and Instagram ads | Splits the audience so each person sees only one version. See our Meta Ads Manager guide |
| Microsoft Clarity | Free heatmaps and session recordings | Finding what to test | Doesn’t run A/B tests, but shows where people get stuck |
How to choose:
- Shopify store? Start with Rollouts for theme-level tests. Add an app only if you need to test individual elements, prices or offers.
- WordPress site? A plugin like Nelio can work. Check it works with your caching plugin, or everyone may get the same version. Still choosing a platform? See our WordPress vs Shopify comparison.
- Most traffic comes from Google Ads? Test landing pages through Google Ads experiments first. Our Google Ads optimisation checklist shows where experiments fit.
- Several sites, regular testing, some developer time? A dedicated platform (VWO, Convert, AB Tasty, Optimizely) is worth comparing on price at your traffic level.
Watch the flicker. Client-side tools load the original page, then swap in the variant. If the script loads late, visitors glimpse the original first, which can bias the result. Install the snippet high in the page head as the tool’s documentation recommends, and check on a slow mobile connection.
Common A/B testing mistakes
| Mistake | What goes wrong | What to do instead |
|---|---|---|
| Peeking and stopping early | Checking daily and stopping when B looks ahead makes false winners far more likely than the 5% you set | Fix sample size and end date before launch. Look at the result only at the end, or use a tool built for sequential testing |
| Too many variants | Splitting 2,400 visitors across four versions gives each one 600, so nothing reaches significance | One variant against the control. Add more only with much more traffic |
| Changing things mid-test | Editing the page, the ads or the offer halfway through spoils both groups | Freeze everything that touches the test page until it ends |
| Ignoring device split | A variant wins on desktop and loses on mobile, and the total hides it | Check the result by device afterwards, and consider mobile-only tests |
| Forgetting to ship the winner | The test ends, the tool is switched off, and the old page comes back | Hard-code the winning version into the site |
| No record of results | Six months later nobody remembers what was tested | Keep a test log (template below) |
A/B test plan template
Copy these fields into a spreadsheet, one row per test, and fill them in before launch.
| Field | Example (tuition centre trial page) |
|---|---|
| Test name and ID | T-07 Trial page headline and form |
| Page or URL | /free-trial-class/ |
| Evidence | 70% of mobile visitors don’t reach the form. Parents’ top WhatsApp question is “which level and subject?” |
| Hypothesis | Changing the headline to “Free Sec 3 and 4 Maths trial class in Bishan” and cutting the form from six fields to three will increase trial bookings, because parents see their child’s level immediately and type less |
| Control (A) | Current headline “Helping every child reach their potential”, six-field form |
| Variant (B) | New headline, three-field form (parent name, mobile, child’s level) |
| Primary metric | Trial booking form submissions |
| Secondary metrics | WhatsApp button clicks, bookings by device |
| Guardrail metric | No-show rate for booked trials (from centre records) |
| Audience and split | All visitors, 50/50 |
| Baseline rate | 4.0% |
| Minimum detectable effect | 50% relative (to 6.0%) |
| Sample size per version | About 1,900 |
| Planned start and end | Mon 4 Jan to Sun 21 Feb 2027 (7 weeks) |
| Tool | Nelio A/B Testing (WordPress) |
| QA done | Both versions checked on iPhone and Android, form tracking checked in GA4 DebugView |
| Result | (fill in at end: rates, lift, confidence interval) |
| Decision and next test | (ship, discard or retest, plus what you learned) |
The guardrail metric is worth keeping. A shorter form might bring more bookings but more no-shows, which would make the “winner” a loss for the business.
How A/B testing fits into a wider CRO programme
One test answers one question. The value comes from a steady sequence where each test builds on what the last one taught you. For ideas to work on between tests, see how to increase website conversion rate.
Pages built with testing in mind (one goal, clear sections, editable headline and CTA blocks) are far easier to test than pages hard-coded by a developer who has since left. Our landing page optimisation guide and these landing page design examples show structures that test well.
Frequently asked questions
How much traffic do I need for A/B testing?
It depends on your conversion rate and how big a change you want to detect. As a rough guide, if a page gets fewer than about 1,000 visitors a week and converts at a few percent, you can only detect large changes (30% or more) within two months. Use a sample size calculator before starting.
How long should an A/B test run?
Until it reaches the sample size you calculated, and for at least two full weeks so every day of the week is covered. If you haven’t reached the sample size after about eight weeks, call it inconclusive and test something bolder.
Is there a free replacement for Google Optimize?
Not a direct one. Google ended Optimize in September 2023. Free options are limited: Shopify Rollouts is included in Shopify plans from Basic upwards, Google Ads and Meta Ads Manager have built-in experiments for ad traffic, and some WordPress plugins have small free tiers. For regular website testing, most businesses pay for a tool.
Does A/B testing affect SEO?
Not if you follow Google’s guidance: no cloaking, a canonical tag on variant URLs, 302 redirects for split URL tests, and end the test once you have a result.
Can I A/B test with WhatsApp as the conversion?
Yes. Track clicks on the WhatsApp button as a GA4 key event and use that as your metric. Using a different pre-filled message for each version also lets you count which chats came from which version.
Need help with testing?
If you’d like a second pair of hands, our conversion rate optimisation team can audit your tracking, build a test backlog from your own data and run the tests. If the page itself needs rebuilding first, our landing page design service builds pages with testing in mind, and our Google Ads agency team can run landing page experiments on your paid traffic. Talk to us about which page to test first.
