Statistical significance
Statistical significance means the probability that a test’s observed difference arose by chance alone is below a threshold set in advance, commonly 5 per cent. It does not show that the difference is large or commercially important. Stopping a test the moment the result looks good raises the chance of a false positive.
- Area
- Data and CRO
- In Turkish
- İstatistiksel anlamlılık
- Read more · Service
- Conversion rate optimisation (CRO)
Why it matters
Ending a test the moment the result looks good leads to mistaking a chance difference for a real one. The threshold is chosen in advance and the test is not stopped before the planned time is up. A significant difference can still be small. The commercial value of the change should be calculated separately.
Example
Say at the end of a test, version B looks 4 percent better than A and the chance that the difference is due to luck comes out at 3 percent. With a 5 percent threshold, the difference counts as significant. But if it amounts to 6 orders a month and the change takes effort, significance alone is not enough.
Related terms
- A/B testAn A/B test shows two versions of a page or element in the same period, assigns visitors to them at random and measures which performs better.
- Conversion funnelA conversion funnel is the ordered view of the steps a visitor takes on the way to a goal, such as product page, basket, checkout and order.
- Conversion rateConversion rate is the share of visits or visitors that complete the intended action: conversions divided by visits.
- CROCRO is the systematic work of getting more of a site’s visitors to complete the intended action, such as buying or sending a form.
- HeatmapA heatmap is a visual report that uses colour to show where visitors click on a page, how far they scroll and where they linger.
- Sample sizeSample size is the number of visitors or conversions each version must see for a test to give a reliable result.
The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch
From our journal
From what we have written on Data and CRO; they show how the term turns up in the work.
- How to build GA4 audiences and use them in Google AdsFrom conditions to sequences, from membership duration to predictive audiences: the steps for building a GA4 audience and what must be on for it to reach Google Ads.
- How to improve on-site search in an online shopFrom measuring search in GA4 and reading the terms that return nothing, to synonym and typo rules, the results page and the “no results” page, in order.
- How to write a GA4 measurement plan: events and key eventsStart from business questions, choose GA4 events, write down naming rules and limits, mark the key events, and put it all in a one-page table.
- What is server-side tagging, and who needs it?How a server container differs from tags in the browser, what it does and does not solve, who should bother, and the steps of a first setup.