Standard cut-offs are not recommended when determining a suitable effect size for a power analysis. Indeed, the ‘meaningfulness’ of an effect size will depend on some subjective elements. That is, a ‘small’ effect may have drastic implications in certain contexts, while ‘large’ effects may have little to no implications in other contexts. A recent publication has provided a practical example to help differentiate a statistically versus a clinical meaningful effect size. I borrow some of the ideas from this paper to help us understand this difference.
Recall Cohen’s published cut-offs that are often misused for determining the magnitude of an effect.
| Guideline | Cohen’s d |
|---|---|
| S | 0.2 |
| M | 0.5 |
| L | 0.8 |
Does this inherently mean that small effects are indeed, small? Well, relative to medium and large, yes. However, even small effects can have major practical and clinical applications.
Adapting from the work of Carey et al., let’s consider the hypothetical incidence of pre- and post-pandemic adolescence depression referrals to mental health professionals in Corner Brook, NL. Let’s assume of population of 20,000; let’s also assume that about that about 10% of these residents are between 10- and 18-years old.
Imagine we could hypothetically give every individual a depression questionnaire as an assessment tool and those with a set cut-off would get a referral to the mental health clinic in the city. The Mood and Feelings Questionnaire (MFWQ) is a suitable assessment and a score of >=12 indicates an increased probability in current depressive psychopathology. The MFQ has a minimum and maximum score of 0 and 26, respectively, mean of 4.92, and standard deviation of 4.49 (note: these are not based on Corner Brook data; rather, mean and SD reported from a different study). Assuming this is the case for pre-pandemic data in Corner Brook, let’s visualize the distribution.
In our data, 186 cases would get a referral. Let’s run a simulation of 500 samples to get a better estimate.
If we take the mean of the 500 simulations, we get an estimate of 199.29 referrals. Given the few child and adolescent mental health professionals in the city, it likely represents a strained system given pre-pandemic levels.
What would a small increase look like?
Assume that the pandemic has led to a small increase in depression in children and adolescents (effect size: \(d=.2\)). We can use maths to determine the hypothetical post-pandemic mean on the MFQ. Here, \(\bar{x}_1\) represents the pre-pandemic mean, \(\bar{x}_2\) the post-pandemic mean, \(SD_p\) is the poled standard deviation (assume SD is the same for pre- and post-). Note: we will calculate for an effect of \(d=-0.2\), because \(\bar{x}_2\) is hypothesized to be larger.
\(d=-0.2=\frac{\bar{x}_1=\bar{x}_2}{SD_p}\)
rearranging:
\(d=\frac{\bar{x}_1=\bar{x}_2}{SD_p}=-0.2\)
\(4.92-\bar{x}_2=(-0.2)(4.49)\)
…
\(-\bar{x}_2=(-0.2)(4.49)-4.92\)
…
\(-\bar{x}_2=(-0.2)(4.49)+4.92=-5.82\)
Thus:
\(\bar{x}_2=5.82\)
How many increase referrals should we see under this new mean? Let’s plot the two figures together. The blue represents the pre- data and the clear bars the post- data.
Again, this is just one simulation; let’s run 500 like before.
If we take the mean of the 500 simulations for post-pandemic data, we get an estimate of 246.11 referrals. What’s the difference?
| Pre | Post | Difference |
|---|---|---|
| 199.286 | 246.11 | 46.824 |
Given our population, we would see an extra 46.824 referrals. This may seem small; but, please, consider the limited mental health professionals in the city. This would mean approximately 3 extra full time psychologists, given a case load of approximately 20 clients. Thus, even a small effect size indicates a significant mental health professional concern.
Thanks to Carey et al. for inspiring this post.
Note: I am not suggesting the pandemic has increased depression. This is purely hypothetical example. Despite this, this demonstrates the potential for ‘small’ effect sizes to have major practical implications. Context matters.