Minimum Detectable Effect(MDE) 🤔

Prerequisite: A/B testing, Sample size calculation

This short article will explain about what is the MDE with an example and how does it help in sample size calculation.

As the name suggests MDE is the minimum change that you want your experiment/test to detect. It is an important factor in sample size calculation and is inversely proportional to it.
Let me explain this with an example. Suppose you want to test a new drug for dogs that whether it can be used for increasing their height or not. And let’s say there are two cases, where you are saying that:

1. the increase in height should be 1cm(MDE).
2. the increase in height should be 10cm(MDE).

In the 1st case since the change is very small hence to justify that it has not happened by chance, you will need more samples. (In the statistics world, we say you will need a high sample size to make the result statistically sound).

On the other hand in the 2nd case since the proposed change in height is large you can be confident even with the smaller sample size that the change is due to the drug.

So with the above example, we can say that MDE is used to measure the experiment sensitivity. To detect the smaller change(high sensitivity) you will need a bigger sample size and to detect the bigger change(low sensitivity) you will need a smaller sample size. So mathematically you can say that:

MDE = K/Sample-size = K'/sensitivity-of-experiment

You can have a doubt here that why not always have the low MDE for the experiments. The answer is, you can have but… low MDE comes at some cost: you will have to run the experiments longer to reach the bigger sample size, and hence you will have to spend more money to achieve that traffic. Like in the drug example, you may need a bigger infrastructure to handle a larger sample size, and a more precise and accurate instrument to measure that small change.

MDE is not something which we calculate, we propose or decide MDE for a particular experiment and it can be different for different experiments, like you can see here how the MDE is different for experiments on different pages of a website.

I hope this will help you understand the MDE better. If you have further questions related to MDE you can ask in the comment section.

Happy Testing :)

References

1. https://splitmetrics.com/resources/minimum-detectable-effect-mde/
2. https://www.brooksbell.com/resource/blog/minimum-detectable-effect/?__cf_chl_managed_tk__=pmd_178c32e2b94cb5c2a592fe03baa8f6beedaf1336-1628425870-0-gqNtZGzNAuKjcnBszQmO
3. https://www.youtube.com/watch?v=BHGB8WS0XbI&ab_channel=BrianDavidHall
4. Thanks to Divyatman Khare