In a factorial ANOVA there may be many effects estimated. Should I use a correction to control the Type I error rate?
This discussion assumes that the data were analyzed using effect coding (as opposed to dummy coding), and that the individual experimental conditions n’s are at least approximately equal across experimental conditions.
In our view, whether to use a correction to control the Type I error rate depends largely on whether you are operating from a conclusion-priority or decision-priority perspective (see Chapter 3 in Collins, 2018). If your purpose is classical null hypothesis testing for the purpose of drawing scientific conclusions about main effects and interactions, you are probably operating from a conclusion-priority perspective. In this case you may wish to use a correction.
Corrections to control the Type I error rate may be particularly important when effect estimates are correlated. It may be helpful to note that whereas dummy coding typically produces substantially correlated effect estimates, effect coding produces estimates that are uncorrelated in the case of equal n’s, or moderately correlated unless the n’s vary widely.
If your purpose is deciding on which components and component levels to select for inclusion in the optimized intervention, you are probably operating from a decision-priority perspective. Here we do not recommend a correction, for two reasons. First, we recommend either not using classical hypothesis-testing at all when working from a decision-priority perspective, or using it as a guiding framework rather than a strict set of rules. (In fact, a Bayesian approach would be more consistent with the decision-priority perspective; we are working on such an approach.) Second, corrections to control the Type I error rate can increase the Type II error rate to unacceptable levels.
All Frequently Asked Questions
Do you see MOST intersecting with implementation science, and if so, how?
Can I expect reviewers of grant proposals to understand MOST? If not, how should I handle this in the application? There is not enough room to provide a lot of background.
I work in the prevention field, in which the horizon on the outcome is very far away. How can I optimize under these circumstances? It seems like it would take decades.
Should I apply for funding to support all three phases of MOST?
How can I find out what other studies using MOST have been conducted in my area?
How can I obtain the background I need to write a grant proposal involving MOST?
Whether you are looking for additional support as you prepare a grant proposal involving MOST or practical information helpful in managing your optimization trial, this section provides resources for a deeper dive into intervention optimization.
REDCap with Most
The goal of this manual is to show how one might setup a REDCap project to support a research study with multiple conditions, such as factorial experiments common in the Multiphase Optimization Strategy (MOST) framework.
Establishing a conceptual model and understanding optimization
Sometimes the conceptual model is not a model of a health behavior per se, but a model of maintaining treatment fidelity, promoting adherence or compliance, or the like. The conceptual model is explained in more...
Common misconceptions about factorial experiments
A factorial experiment is essentially an RCT with a lot of experimental conditions, and therefore is extremely difficult to power.
Informal introduction to factorial experimental designs
The purpose of this page is to clarify some concepts, notation, and terminology related to factorial experimental designs, and to compare and contrast factorial experiments to randomized controlled trials (RCTs). A more in-depth introduction can...
LET’S STAY IN TOUCH
Join the cadio Mailing List
Keep up to date with the latest news, events, online courses, and resources from cadio.