Benjamini-Hochberg beyond FDR
Preamble
Welcome to my blog! This post has been gestating for some time.
I studied multiple testing during my PhD, which in statistics
The key point
Statistical procedures are typically underlied by decision theory.
Unlike single hypothesis testing with p-value thresholds, multiple testing with the BH procedure and (b) FDR control does not have a straightforward utility-maximizing interpretation. In fact, the most straightforward interpretation is that it keeps utility close to zero.
This actually goes not just for BH, but the whole paradigm of “control FDR subject to maximizing number of discoveries”.
A fundamental problem in statistics: compound decision making
Type I error: P(reject i | i true). FDR: P(i true | reject i). If we know the prior and our cost-benefit ratio, is it obvious how to trade off?
null distribution f(p | i true) and local FDR P(i true | p). Here, we will know how to trade off…
Robbins’ theory of compound decision making
An application
You can take hypothesis testing in a game theoretic direction. If you want to guarantee breaking even against agents of some type, you can control FDR.