I have a question, and a comment… First the comment: Andrew Gelman and his co-authors have written extensively about ‘garden of forking paths’ (e.g. here http://www.stat.columbia.edu/~gelman/research/unpublished/p_hacking.pdf). Their point is that p-values can be invalid even if the researcher would not have actively done any “p-hacking”. Already the fact that a researcher adjusts her analyses (data transformations, outlier filtering, choice of statistical test, etc…) according to the data she has at hand invalidates the theoretical interpretation of p-values. This is because the analyses are dependent on the data; if the data were different, the analyses might also change! This seems like a profound problem in all empirical work, and also seems very poorly understood.

Then the question… After reading ASA’s (and your) warnings, should I stop using p-values? Wouldn’t many of the alternatives proposed by ASA like bayes factors, likelihood ratios, (bayesian) standard error bands, etc. suffer from many of the same problems? If I can ‘hack’ a p-value, I should be just as able to hack a bayes factor, right? In addition, when it comes to large datasets, I might also encounter some parameter estimates which have a very large bayes factor, but the parameter estimate could still too small to be of any practical relevance, or could be a result of an invalid research design.

In addition, if we move the goalposts so that instead of p<0.05 we would set the threshold of "significant" to p<0.01 or some other nunber smaller that 0.05, wouldn't that just encourage hacking in cases when a result is close to p=0.01?

I admit that this is more a question to ASA than you guys, but ASA does not have such a nice blog as you have. 🙂

Here a brief interest of write up about your article as my PhD research study is similar on Internet related issues among school students.

Warm regards

Mary

What would be really interesting would be to compare this data with a real measure of internet connectivity—say, unique IP addresses per household—and see which areas are more or less connected than their demographic information would predict.

I read your post with enthusiasm, I do agree with your view on how social scientists should evolve their skills. I am a sociologist and I work in the agent-based modeling field, so I had to pick up a lot of computer sciences skills in order to manage data and run computational models.
When I was working on my PhD, it was hard to find other sociologists who were skilled in computer science (at least in Italy), so I learned mainly by asking questions on mailing lists, reading books and by errors.
After 8 years I think a have mastered a lot, and occasionally I have the chance to give some of this back to young students… but more is needed and Social Sciences Departments ought to deal with this fact: students need to learn how to program, other than run statistical softwares.

Best regards,
Simone

Thank you

Tony, thanks for your comment. On the hit rate, you are correct, it does increase it, but only minimally. Most commonly, the web-crawler visits the website once. We may visit another time for validation purposes. These few visits are unlikely to increase their stats significantly. But this is something to keep in mind for future research on these issues.

The one concern I have is that if the crawlers mentioned in the article are visiting these illegal sites, doesn’t that increase their hit rate and encourage more offenders to either contribute to existing sites or start up their own? Both activities that would risk more child exploitation happening.