The problem of demand effects in power studies: Moving beyond power priming

Abstract

Power in experimental research has been commonly induced by methods that raise concerns regarding demand effects. In this paper, we investigate the empirical relevance of these concerns. In an incentivized online study (N = 1632), we manipulated the method of power manipulation (power priming vs. resource allocation), the level of power (high-power vs. control), and the presence of a manipulation check after the power manipulation. We then assessed risk-taking as an outcome variable in two ways, once as a non-consequential measure (self-report measure) and twice as a consequential measure (incentivized behavioral choices). Our results show that both using power priming (vs. resource allocation) and implementing a manipulation check substantially increased the potential for demand effects as measured by the proportion of participants who were aware of the study hypothesis. In addition, we were able to replicate the positive effect of power on risk-taking previously reported in the literature. However, we only found a significant (and small) effect for our non-consequential measure of risk-taking; when risk-taking was measured with either of our two consequential measures, power had no significant impact. Our pattern of results shows that concerns about demand effects in priming studies cannot be dismissed. We advise researchers, especially those studying power, to steer away from demand-prone manipulations of power and to measure outcome variables (e.g., behavior) through consequential choices.