A/B Testing
An A/B test cannot tell you what to change. A test only reveals which of two guesses was less wrong. Ivan Sp built a custom WordPress A/B testing plugin after existing options fell short. They required rebuilding pages inside a proprietary designer, or paying an ongoing subscription for something conceptually simple. A live test on the finished tool produced a measurable lift in demo bookings.
Why an Existing Plugin Was Not the Answer
Most WordPress A/B testing tools take over page building entirely. A page has to be rebuilt inside the plugin’s own editor before it can be tested. Others charge a recurring subscription for functionality that is conceptually simple. Neither fit a workflow built around real pages already living in a normal WordPress editor.
The custom plugin avoids both tradeoffs by design. No page needs rebuilding in a proprietary designer. No subscription runs in the background once it is installed.
A Live Test With a Measured Result
The plugin has run a live test producing a measurable lift in demo bookings, not a theoretical capability sitting unused. That evidenced result is why this practice earned the Conversion Rate Optimization name, rather than a more conservative label suited to an untested program.
Where A/B Testing Fits in a Broader CRO Engagement
Testing answers one narrow question well: does version A or version B convert better on a specific, already-identified element. Deciding what is worth testing happens earlier, covered under landing page and trust audits and heatmaps and session recording.
A test run on a page nobody has diagnosed usually tests the wrong variable. Diagnosis comes first, and the test then confirms or refutes a specific, evidence-based hypothesis rather than a guess.
Where the Standard Testing Advice Breaks Down
- The usual advice
- Watch the dashboard, and call the test as soon as one version pulls clearly ahead.
- What holds up in practice
- Stopping when a result looks good is the fastest way to manufacture a false positive. Early leads reverse routinely, and the stopping rule has to be written before the test starts.
Picture a marketer who checks a test on day four and sees version B ahead by 89 percent. The result is real arithmetic on a small sample.
The test is called, the change ships, and the conversion rate does not move. By the planned stop the true difference was 22 percent, and the reported figure was four times that.
I write the stopping rule first and read the result once. A null result recorded honestly is worth more than a win that cannot be reproduced.
Over-reported wins set an expectation the next test cannot meet. The credibility of the whole programme is spent on a number that was never there.
Four Things Commonly Assumed About A/B Testing
Was your last test stopped on a date, or on a number that looked good?
I meet these four assumptions whenever a testing programme produces unreproducible wins.
| Commonly assumed | What actually holds |
|---|---|
| A clear early lead means the test is decided. | Early leads reverse routinely. Stopping when a result looks good is the fastest way to manufacture a false positive. |
| More visitors means a faster result. | Sample size is driven by conversion count rather than visitor count. A high-traffic page with few conversions still tests slowly. |
| Testing several changes at once saves time. | Two changes at once make the result unattributable. Neither change can be credited or ruled out. |
| A null result is a failed test. | A null result is an answer, and recording it prevents the same idea being retested in a year. |
| Split testing risks a cloaking penalty. | Google publishes guidance on website testing. Avoiding cloaking, using temporary redirects, and running only as long as needed is the whole requirement. |
An over-reported win sets an expectation the next test cannot meet. The credibility of the whole programme gets spent on a number that was never there.
What a Valid Test Requires
- One variable
- A single changed element. Two changes at once make the result unattributable.
- A stable split
- Consistent assignment, so a returning visitor sees the same version each time.
- Real published pages
- Both versions live and indexable, rather than assembled in a builder overlay.
- Enough conversions
- Sample size is driven by conversion count, not by visitor count.
- A pre-stated hypothesis
- The element, the change, and the expected direction, written before the test runs.
- A fixed stopping rule
- Set in advance, since stopping when a result looks good inflates false positives.
- The measurement surface
- GA4 key events, recorded per variant.
- Why a custom build
- Google Optimize stopped serving experiments on 30 September 2023.
A Worked Example: Reading a Test Honestly
Consider a test on a demo booking headline. The two readings below are illustrative, written to show how the same data supports opposite conclusions. No client engagement is described here.
| Measure | Read at day 4 | Read at the planned stop |
|---|---|---|
| Visitors per variant | 310 | 2,940 |
| Conversions, version A | 9 | 121 |
| Conversions, version B | 17 | 148 |
| Apparent difference | 89 percent better | 22 percent better |
| Confident the difference is real | No | Yes |
Ivan Sp reads the test at the planned stop rather than at day four. The early number was real arithmetic and a misleading conclusion.
Stopping a test when it looks good is checking the score at half time and cancelling the second half. The scoreboard was accurate, and the match was not over.
The honest lift here is 22 percent, which is still worth having. Reporting 89 percent would set an expectation the next test cannot meet.
How the Test Actually Runs
I built the plugin to rotate traffic between two real, already-published pages, splitting visitors between the versions compared. Conversion events get measured by tracking specific CTA element IDs directly, rather than a third-party analytics layer bolted onto the page.
Tracking the exact button or link a visitor clicked ties the result to the action that matters. A broader, noisier session-level metric would not show that connection as clearly.
How a Test Runs
I write the stopping rule before the test starts, which is the step that keeps the result honest.
- Confirm the GA4 key event fires correctly on both versions before any traffic is split.
- Write the hypothesis as one sentence naming the element, the change, and the expected direction.
- Estimate the conversions needed, based on the current rate and the smallest lift worth acting on.
- Publish both versions as real pages, so neither variant is a rendered overlay.
- Split traffic evenly, and assign each visitor consistently across return visits.
- Set the stopping rule and the run length in advance, then record both.
- Leave the test alone while it runs, resisting the pull to read partial results.
- Read the result once at the planned stop, and accept a null result as a real answer.
- Implement the winner, and archive the losing version rather than deleting the record.
- Record every test, including the failures, so the same idea is not retested in a year.
How Testing Work Is Scoped
I scope testing against conversion volume, which decides how long each test must run.
- What arrives
- A hypothesis, a running test, a stated stopping rule, and a written result including null results.
- Typical duration
- Three to eight weeks per test, driven by conversion volume rather than by effort.
- What the client supplies
- GA4 access, site access, and agreement on the smallest lift worth acting on.
- Included
- Tracking verification, hypothesis writing, test setup, monitoring, and honest reporting.
- Not included
- Design and copywriting for the variant, and any promise that a test will find a winner.
- What changes the estimate
- Low conversion volume, seasonal traffic, or pages behind a login.
- How it is quoted
- After a short scoping call, once conversion volume is known.
Tools and Primary Sources
The references below cover measurement and test validity. Each opens in a new tab.
- Google Optimize sunset notice confirms that Google’s own free testing tool stopped serving experiments in 2023.
- GA4 key events documents how a conversion is defined and counted in GA4.
- Google guidance on testing and search sets out how to run a split test without creating a cloaking or duplicate content problem.
- Microsoft Clarity records sessions per variant, which is useful for understanding why a version lost.
Check Whether You Can Test at All
I suggest one calculation before anyone builds a test.
- Open GA4 and find the monthly conversion count for the page you want to test.
- Halve it, since each variant only receives half the traffic.
- Ask whether that halved number would let you notice a difference of a fifth either way.
- If the answer is no, note how many months of running it would take.
Fewer than roughly a hundred conversions per variant makes most tests inconclusive. At that volume, a trust audit and usability sessions return more than a test would.
Common Questions
What kind of elements typically get tested first?
Primary call-to-action buttons and headlines above the fold usually get tested first, since they carry the most traffic and the clearest, most directly measurable outcome.
How long does a test need to run before results are reliable?
It depends on traffic volume. A page needs enough visitors split between versions to reach a statistically meaningful sample, which can take days or several weeks.
Does the plugin work on any WordPress theme?
The plugin works with standard WordPress pages and CTA elements carrying a trackable ID. A quick compatibility check confirms fit before a test gets set up on a specific site.
Does running an A/B test risk a search penalty?
Not when it is run properly. Google publishes guidance on website testing with three requirements. Avoid cloaking, use temporary redirects rather than permanent ones, and run the test only as long as needed.
Discuss This Work
The first call is a free 30-minute consultation with no pitch attached. You receive an honest assessment of where the site stands and what can realistically be built.