Statistical significance

Statistical significance means the probability that a test’s observed difference arose by chance alone is below a threshold set in advance, commonly 5 per cent. It does not show that the difference is large or commercially important. Stopping a test the moment the result looks good raises the chance of a false positive.

Why it matters

Ending a test the moment the result looks good leads to mistaking a chance difference for a real one. The threshold is chosen in advance and the test is not stopped before the planned time is up. A significant difference can still be small. The commercial value of the change should be calculated separately.

Example

Say at the end of a test, version B looks 4 percent better than A and the chance that the difference is due to luck comes out at 3 percent. With a 5 percent threshold, the difference counts as significant. But if it amounts to 6 orders a month and the change takes effort, significance alone is not enough.

The definitions are general information. If you think one is incomplete or wrong, write to us and we will reply within one working day. Get in touch

From our journal

From what we have written on Data and CRO; they show how the term turns up in the work.

Tell us about
your project.

Fill in the form or write to us directly, and we will agree on a day to talk.

Your first meeting is with Deniz Sezer, founder of Detmore.

First meeting free · Reply within one working day · Written proposal in 2 to 3 days · NDA ready

What do you need? pick as many as you like

Name, email and phone are required.

Contact

We reply within one working day.