-
Notifications
You must be signed in to change notification settings - Fork 5
Description
I had an interesting discussion today at lunch with Dave Mathews that I thought might deserve discussion here. We were talking about the way that analysis of MD simulations is done iteratively, and he pointed out that this is the way p-hacking occurs.
Imagine we have two related simulations (or sets of simulations) we're comparing: we try one thing, and the p-value says they're not significantly different, so we try another, and another, and we focus on the quantities that say the two simulations are different. How is that not p-hacking? I know there are ways to handle this correctly (I think you have to essentially lower the p-value for significance with each successive test, though I don't know the theory), but I've never heard of anyone actually doing this in the context of MD.
Does someone with a stronger stats background than mine have a suggestion for what the best thing to do is?