Topic: Replication of research results

[Citizen Skeptic]

Interesting podcast on Econtalk on this subject.

http://www.econtalk.org/

Ed Yong, science writer and blogger at "Not Exactly Rocket Science" at Discover Magazine, talks with EconTalk host Russ Roberts about the challenges of science and science journalism. Yong was recently entangled in a controversy over the failure of researchers to replicate a highly-cited and influential psychology study. He discusses the issues behind the failed replication and the problem of replication in general in other fields, arguing that replication is under-appreciated and little rewarded. After a discussion of the incentives facing scientists, the conversation turns to the challenges facing science journalists when work that is peer-reviewed may still not be reliable.

[Skeptress]

I think journalists in general these days are under-educated in their fields (ex. science) and more likely to go for the sensational story without fact checking. 

[Citizen Skeptic]

It's a pretty good podcast. I didn't know the average lifetime of NFL players wasn't 58.

[Shadow Of A Doubt]

Ed Yong writes a great blog here.

Listening to this now!

[Zytheran]

I work as a scientist. I routinely suggest replicating certain experiements, and then add a variation to make it relevant to our clients.
It have been repeatedly told, don't bother, it will never be published. The journals very rarely publish replications and never just direct replications in my experience. (Why would they anyway when you can publish something new which will be more interesting?)
Makes you want to hope the first people to publish always get it right...
So yes "replication is under-appreciated and little rewarded."

[Citizen Skeptic]

Since peer review doesn't seem to be sufficient, how about requiring replication by at least one independent/unrelated group?

It seems like bad practice to publish a single result that hasn't been replicated now that I think about it.

[Zytheran]

Since peer review doesn't seem to be sufficient, how about requiring replication by at least one independent/unrelated group?

It seems like bad practice to publish a single result that hasn't been replicated now that I think about it.

Sorta gets covered by having stats to calculate the p value, however with a p of 0.05 1 in 20 bogus results will get through.
Also having a detailed explanation of exactly the results are derived can show that many bad things didn't happen. However I notice that there are very few papers that have enough information to allow me to replicate the results without first needing to ask the authors a few questions.

[JuniorSpaceman]

Sorta gets covered by having stats to calculate the p value, however with a p of 0.05 1 in 20 bogus results will get through.
Also having a detailed explanation of exactly the results are derived can show that many bad things didn't happen. However I notice that there are very few papers that have enough information to allow me to replicate the results without first needing to ask the authors a few questions.

I'm not a scientist, but I find this incredibly depressing as someone who tries to spread the message on science and skepticism. It really seems to me that science itself is broken, if the most powerful thing about it (its self correcting mechanism through review and replication) never actually happens in the real world.

[Citizen Skeptic]

Science is not broken. Trust science. Don't trust scientists.

The hard part is separating what is really known, from what we think is really known.

Would you throw physics out the door if they discovered that string theory went nowhere and that everything would have to be thought over, starting with the standard model and quantum mechanics? I would not. I would label it progress. Many would label it failure. I think that's the problem.

[Zytheran]

Sorta gets covered by having stats to calculate the p value, however with a p of 0.05 1 in 20 bogus results will get through.
Also having a detailed explanation of exactly the results are derived can show that many bad things didn't happen. However I notice that there are very few papers that have enough information to allow me to replicate the results without first needing to ask the authors a few questions.

I'm not a scientist, but I find this incredibly depressing as someone who tries to spread the message on science and skepticism. It really seems to me that science itself is broken, if the most powerful thing about it (its self correcting mechanism through review and replication) never actually happens in the real world.

Well, scientists are people too  and make mistakes. It's not completely broken but it's not completely good either. There is room for improvement however a lot of poor papers do get picked up and never printed and a few bad ones that get through do get picked up in follow up comments. And even fewer get through and are accepted by everyone and then prove to be wrong. Remember that before a paper gets published it is reveiwed by anonymous scientists so that culls the wheat from the chaff and reduces the chance something is really screwey. And although pure replications don't occur often it sort of occurs when scientists expand upon an idea with a follow up idea. If there was something horribly wrong with the first experiment it gets found by a extension experiment that fails becuase the fundamental assumption about the first experiment isn't born out.
Also remember than many experiments *are* replicated by students in Uni but simply don't get published, they get written up and graded. If students find a problem with an existing experiment there is good chance their findings are publishable if something new has been discovered.
Like everything in the world science isn't black and white, it's a spectrum of greys. There is some poor science and there is lots and lots of good science.
My main point is not to preach that science is perfect or a perfect solution, it's simply the best way to increase our knowledge we know of.
So, don't give up hope, just be reasonable with your expectations of what is practicable and reasonable in a market driven economy.

[JuniorSpaceman]

(I had such a great response here and then my browser crapped out, so here's the summary )

CS, I agree - I believe in science if I believe in anything. I just want what's 'done' to be science at its best.

Zytheran, I also agree - people are fallible, but the reason that science is better than nonsense is because of review, replication, etc. A single study is something like an anecdote - worthy of investigation and hypothesis development, but only a starting point. It's bad enough that corporations use science in a bad (I don't necessarily mean morally, but rather methodologically) way - I'd like to know that we're pushing forward towards the truth.

OK, here's an idea - sure there are a ton of papers that have some incremental impact on our understanding of the world, but pretty much fit into what's understood, but whever a paper comes along that challenges something that we believe, that study should be focussed on for replication ahead of new research.

[ting-bu-dong]

Science is not broken. Trust science. Don't trust scientists.

Seconding the notion that it is crucial to distinguish between individual publications and the literature as a whole. The mistake a lot of people, in particular the media, make is to focus on one groundbreaking paper while ignoring the rest of the work done on the same subject. It's a bit like trying to predict the outcome of an election by polling one voter.

In the physical sciences, which are the only ones I have first hand experience with, a publication is not the ironclad last word on a question. It is a preliminary result of work done by a particular group. Replication is not much of an issue in that case because a given question will be investigated from similar angles by multiple groups who each publish their results as they go along, and the results of one group influence and put in context the result of all the others, even though the experiments the different groups do won't be identical and won't strictly be replications of each other. You don't need to replicate a finding if you are using a different methodology to study the same phenomenon because your results should be the same. And figuring out why they aren't is part of the process.

[Citizen Skeptic]

In the physical sciences, which are the only ones I have first hand experience with, a publication is not the ironclad last word on a question. It is a preliminary result of work done by a particular group.

This is a good point. Very few papers are definitive and very few results important.