The result was not significant. Was it wasted effort?

No. Learning that a change is not worth 20% or more is actionable, because it stops you investing further in that direction. The wasted tests are the ones with no hypothesis and no stopping rule.

How to Run A/B Tests That Aren't Just Self-Deception: Sample Size, SRM Checks, and Choosing Between GA4, PostHog and GrowthBook

The "41% Lift" That Proved Nothing

A client sends a screenshot: new CTA at 3.1%, old one at 2.2%, "can we ship it?" We asked three questions. How many days? Four. Users per arm? 1,180 and 1,460. How often did you check? Every morning. Each answer maps to a fatal flaw: insufficient sample, sample ratio mismatch, and false positives from repeated peeking. The expensive part came later: the team rolled the change sitewide, revenue didn't move three months on, and nobody could explain why.

When A/B Testing Fits, and When It Doesn't

Good fit	Poor fit
≥ 1,000 conversion events per month	< 200 per month, so any test runs half a year
One north-star metric and a prior hypothesis	"Let's see which looks better," no hypothesis
Page and environment stable for the duration	Mid-redesign, payment migration, or a big promo
Engineering capacity for server-side or SDK bucketing	Third-party JS only, while caring about Core Web Vitals
B2C, e-commerce, SaaS signup funnels	B2B with five high-ticket deals a month

For low-traffic sites the right move is a CRO audit plus qualitative research: 5–8 moderated usability sessions, Microsoft Clarity replays and heatmaps, step-by-step checkout drop-off analysis. At small sample sizes these carry far more explanatory power.

Alternatives Matrix

Option	Bucketing	Statistics	Starting cost (NT$)	Main risk
GA4 + BigQuery, DIY	Roll your own flags	Python/R, self-computed	Tool 0; queries 500–3,000/mo	You build the stats pipeline; event latency
PostHog	SDK, client or server	Frequentist plus sequential	3,000–15,000/mo past the free tier (pricing)	Cost scales sharply with events
GrowthBook	Flags, fully server-side capable	Bayesian plus sequential (docs)	Self-host 0, hosting 1,500–4,000/mo	You own the SDK endpoint and warehouse
Statsig	SDK plus server-side	Sequential, CUPED	Large free tier, then per-event	Advanced features gated to enterprise
VWO / Optimizely	Visual editor	Built in	From ~150,000/yr; Optimizely often seven figures	Client-side injection: highest flicker and CLS risk
Google Optimize	—	—	—	Sunset September 2023 (notice), don't plan around it

The Full Process (5 Phases, 6–10 Weeks)

Phase 1 — CRO audit and hypothesis backlog (1–2 weeks). Deliverables: funnel drop-off table, replay summary, 20–30 hypotheses ranked by ICE. Tools: GA4 Explorations, Microsoft Clarity, Figma, Notion.
Phase 2 — Design and sample size (3–5 days). Deliverable: a spec with one primary metric, three guardrails, MDE, duration, stopping rule. Tools: Evan Miller's and GrowthBook's calculators.
Phase 3 — Implementation (1–2 weeks). Deliverables: flag service wired up, server-side bucketing (Next.js or Laravel middleware), assignment ID in a first-party cookie, SRM dashboard.
Phase 4 — Run and monitor (3–7 weeks). Deliverables: daily SRM check, weekly guardrail report on error rate, LCP and support tickets. The primary metric stays sealed.
Phase 5 — Readout and decision (3–5 days). Deliverables: effect size with confidence intervals, segment breakdown, ship or rollback decision.

Real Cost Breakdown

Item	Amount (NT$)	Notes
CRO audit and hypothesis backlog	From 35,000	25–35 hours
CRO Experimentation Starter Package	From 80,000	Flag architecture, server-side bucketing, SRM monitoring, first experiment
CRO Monthly Advisory	From 25,000/month	1–2 experiments per month, designed and read out
Tooling subscription	0–15,000/month	Self-hosted can be zero; SaaS scales with events

Hidden costs:

Warehouse queries. BigQuery bills past the free 1 TB monthly scan; NT$500–3,000/month is common for high-event sites.
Technical debt. Uncleaned flags become a maintenance burden within six months (4–8 hours per quarter).
Performance tax. Client-side anti-flicker snippets block rendering and worsen LCP and CLS, where the good threshold is 0.1. The rework costs more than doing it right.
Opportunity cost. A seven-week experiment freezes that page for seven weeks.
Analysis labour. 6–10 hours per experiment; an unread result is worth zero.

Client Expectations vs. Reality

What clients assume	What actually happens
"We'll know in two weeks"	At 800 daily visitors and a 2.4% baseline, a 20% relative lift (to 2.88%) needs about 17,500 users per arm, 35,000 total, which is roughly 44 days, rounded to 7 whole weeks. A 14-day run only detects lifts of roughly 35% or more
"More variants, faster answers"	Four arms share the same traffic: 44 days becomes about 88 days, and multiple comparisons push false positives from 5% to 14%
"We need daily numbers"	Daily peeking on a fixed-horizon design pushes the real type I error rate to 20–30%
"A winner means revenue goes up"	Signup rate wins while refund rate quietly rises. Without guardrails you never see it
"Install the tool, insights appear"	Hypothesis quality decides everything. Teams without a backlog run dry by month three

Six Traps and How to Avoid Them

Sample Ratio Mismatch. A 50/50 split landing at 48/52 means bucketing or tracking is broken and the result is void. Fix: daily chi-square test; stop and debug at p < 0.001 (SRM checker).
Peeking and stopping on a win. Fix: hard-code the end date, or use sequential testing with always-valid p-values (GrowthBook and Statsig support it).
Multiple comparisons. Watch five metrics and the chance of at least one "significant" result is 22.6%. Fix: one primary metric, the rest guardrails, plus a Benjamini-Hochberg correction.
Flicker (FOUC) and layout shift. Client-side swaps show the old version first. Fix: bucket server-side or at edge middleware.
Cross-device identity leakage. The same person sees A on mobile and B on desktop. Fix: bucket by user_id when logged in, cookie ID otherwise, with a fixed hashing seed.
Novelty effects and weekly seasonality. New designs look good for three days. Fix: run whole weeks, minimum 14 days.

Success Metrics and the 90-Day Roadmap

Day 30: flags and server-side bucketing live, SRM green daily, ≥ 20 hypotheses in the backlog, first experiment running its full duration. Metrics: no LCP or CLS regression.
Day 60: 2–3 experiments read out including losers, segment breakdowns for new vs. returning and mobile vs. desktop. Metrics: ≥ 80% completion rate, a written conclusion per test.
Day 90: at least one winner shipped and back-tested. Metrics: 5–15% relative lift on the primary funnel, two experiments per month, zero flag debt.

Decision Checklist (12 questions, all yes before you start)

☐ ≥ 1,000 conversion events per month?
☐ Can I name one single primary metric?
☐ Have I calculated MDE and duration, and can I live with it?
☐ Willing to freeze that page for the whole test?
☐ Three guardrail metrics (errors, performance, refunds/complaints)?
☐ Can I bucket server-side or via SDK rather than raw JS injection?
☐ A daily SRM check in place?
☐ Stopping rule and decision criteria written in advance?
☐ Do I accept a null result without moving the goalposts?
☐ A hypothesis backlog to feed the next test?
☐ Do I know who cleans up the flag afterwards?
☐ If the math says three months or more, will I switch to a qualitative CRO audit?

FAQ

My site gets 200 visitors a day. Can I still A/B test?

At a 2.4% baseline, detecting a 20% lift takes roughly 175 days, and the environment will have changed. Redirect that budget into a CRO audit, usability testing and funnel repairs.

Google Optimize is gone. What is the free replacement?

GA4's BigQuery export (free) with self-hosted GrowthBook is the cheapest viable stack, but it needs engineering and statistics capability. Without engineers, use PostHog or Statsig free tiers.

Do I really need server-side bucketing?

If you care about SEO and Core Web Vitals, effectively yes. Client-side bucketing needs an anti-flicker mask that degrades LCP and causes layout shift.

The result wasn't significant. Was it wasted?

No. "This change isn't worth 20% or more" is actionable. The wasted tests are the ones with no hypothesis and no stopping rule.

How does ScriptWalker price CRO work?

CRO Experimentation Starter Package from NT$80,000; CRO Monthly Advisory from NT$25,000/month; low-traffic sites can take the CRO Audit package from NT$35,000.

Next Step

Start with the 12 questions above. If you stall on duration, what you need isn't an experimentation tool but an honest CRO audit. Book a free 30-minute consultation: we'll run the sample size calculation on your actual traffic, then tell you whether to A/B test or go qualitative.