Quis custodiet ipsos custodes?

The Scientific method requires that scientists communicate their findings, clearly, accurately, honestly, and truthfully. Sir Isaac Newton said “If I have seen further it is by standing on ye shoulders of Giants.” Each insight builds upon the framework of countless insights before, and the observations that vouched for their truth. Insight leads to hypothesis, hypothesis leads to predictions, predictions lead to experiments, observations, and measurements, and we compare the predictions to the empirical data and accept, adjust, or discard the hypothesis. Reporting on this cycle of predict, measure, compare, adjust is how scientists disseminate knowledge.

Since no one is infallible, modern scientific scholarship relies on the evaluation of people with similar competence to the original researchers; their peers. The peers review the research report for relevance, quality, and accuracy. This process of peer-review helps expose any flaws and areas that are unclear or need improvement. Research work may be accepted, sent back for revision, or even rejected. Only when a researcher’s peers are satisfied with the quality of the research is the resulting report said to be peer reviewed. It is the gold standard of scientific quality. But the peer reviewers are human as well, and in many cases are in competition with the researchers whose work they are judging. It is a testament to the intellectual integrity of the vast majority of researchers that the peer-review system works as well as it has to date.

A very interesting paper, peer-reviewed!, appeared three decades ago, entitled “Peer-review practices of psychological journals: The fate of published articles, submitted again” I found the results fascinating.

A growing interest in and concern about the adequacy and fairness of modern peer-review practices in publication and funding are apparent across a wide range of scientific disciplines. Although questions about reliability, accountability, reviewer bias, and competence have been raised, there has been very little direct research on these variables.

The present investigation was an attempt to study the peer-review process directly, in the natural setting of actual journal referee evaluations of submitted manuscripts. As test materials we selected 12 already published research articles by investigators from prestigious and highly productive American psychology departments, one article from each of 12 highly regarded and widely read American psychology journals with high rejection rates (80%) and nonblind refereeing practices.

So, what do you think happened?

With fictitious names and institutions substituted for the original ones (e.g., Tri-Valley Center for Human Potential), the altered manuscripts were formally resubmitted to the journals that had originally refereed and published them 18 to 32 months earlier. Of the sample of 38 editors and reviewers, only three (8%) detected the resubmissions. This result allowed nine of the 12 articles to continue through the review process to receive an actual evaluation: eight of the nine were rejected. Sixteen of the 18 referees (89%) recommended against publication and the editors concurred. The grounds for rejection were in many cases described as “serious methodological flaws.” A number of possible interpretations of these data are reviewed and evaluated.

There is a lot of food for thought here. I did a quick check and found a single citation of this paper. I think I’ll dig a little deeper and see if other studies of the peer review process itself have been done over the last couple of decades.

Advertisements

Bert and Ernie visit IceCube

IceCube is a neutrino telescope, consisting of a layer of ice interlaced with photomultipliers more than 1,000 meters below the surface in the Antarctic. Neutrinos are very low mass, ghostly particles which seldom interact with matter. IceCube detects neutrinos by recording Cherenkov photons emitted from secondary leptons produced in very rare interactions of neutrinos with water molecules in the Antarctic’s glacial ice. IceCube began observations in December 2010. Most neutrinos IceCube is expected to see originate in the Sun or cosmic rays interacting with the Earth’s atmosphere.
The first two years of data have been reduced, and two very high energy neutrino events were found (although at a level of statistical significant  is just below the point where they would be considered robust detections). These two events were whimsically named Bert and Ernie. The amazing part for me was the incredible energy of these neutrinos; 1.04 +/- 0.16 PeV and 1.14 +/- 0.17 PeV. That’s Peta eV, a thousand million million electron volts, or the mass equivalent of a million protons.
Image
This energy level means that, as understood today, the neutrinos were unlikely to have originated near the Earth (or as an effect of the GZK cutoff for intergalactic cosmic rays). This is a tantalizing hint that we may have new astronomical neutrino sources out there! But with so few detections–I can’t really call them observations yet–it is hard to say anything with any degree of certainty. I’ve heard of some new results at lower energy which are reported to be statistically significant, but I don’t have any real info. Yet.
Neutrino astronomy is in its infancy, but already we are seeing hopeful signs of new physical phenomena just waiting to be explored.

Billion year old water could harbor early life.

From the National Post.

An international research team reported Wednesday that miners near Timmins are tapping into an ancient underground oasis that may harbour prehistoric microbes. The water flowing out of fractures and bore holes in one mine near Timmins dates back more than a billion years, perhaps 2.6 billion, making it the oldest water known to exist on Earth, says the team that details the discovery in the journal Nature.

This water doesn’t predate life on Earth–we think life first appeared around 3.5B years ago–but it suggests that life may have originated below the Earth’s surface, where it would have been protected from UV radiation from the Sun. Before the evolution of bluegreen algae and the subsequent increase of atmospheric oxygen necessary to form the ozone layer which today protects the Earth’s surface from harmful amounts of UV radiation, life would have been restricted to the oceans. And now, perhaps underground as well. If ancient living organisms are found in the isolated underground water… I find this simply amazing to contemplate.

A flash of insight

Near the end of April, a record setting Gamma Ray Burst was observed in the constellation Leo. Not only did scientists observe the highest energy gamma ray photons ever measured from such an event, about 35 billion times more energetic than photons of visible light, but the duration of the event also set records. The hours-long ‘burst’ enabled other telescopes to observe the region of the sky containing the source object. The redshift distance was remarkably small for GRBs, too. This GRB was exceptionally energetic, remarkably long-lived, and closer to us than 95% of other GRBs seen. The image below is from NASA’s Swift X-ray telescope.

Image

The followup observations are expected to find a supernova remnant by the middle of May. The data collected may reveal new information about the physical processes involved in creating the most luminous events in the Universe.

Observation is the judge of the truth of an idea

It was thought in the Middle Ages that people simply made many observations, and the observations themselves suggest laws. But it does not work that way. It takes much more imagination than that. So the next thing we have to talk about is where the new ideas come from. Actually, it does not make any difference, as long as they come. We have a way of checking whether an idea is correct that has nothing to do with where it came from. We simply test it against observation. So in science we are not interested in where an idea comes from.

There is no authority who decides what is a good idea. We have lost the need to go to an authority to find out whether an idea is true or not. We can read an authority and let him suggest something; we can try it out and find out if it is true or not. If it is not true, so much the worse–so the “authorities” lose some of their “authority.”

Richard Feynman

Science is the belief in the ignorance of experts

Retraction Watch is a blog which “track[s] retractions as a window into the scientific process.” A very interesting RT post came up recently which led to a NYT Magazine article on scientific fraudster Diederik Stapel. Dr. Stapel currently has 53 scientific papers which have been retracted.

A fair number of published, and therefore peer-reviewed, scientific papers have been retracted over the last few years, especially in Medicine. A PNAS study published last year found

A detailed review of all 2,047 biomedical and life-science research articles indexed by PubMed as retracted on May 3, 2012 revealed that only 21.3% of retractions were attributable to error. In contrast, 67.4% of retractions were attributable to misconduct, including fraud or suspected fraud (43.4%), duplicate publication (14.2%), and plagiarism (9.8%). Incomplete, uninformative or misleading retraction announcements have led to a previous underestimation of the role of fraud in the ongoing retraction epidemic. The percentage of scientific articles retracted because of fraud has increased ∼10-fold since 1975.

retract

The paper suggested that the role of scientific misconduct has been underestimated because the retraction notices often downplay or hide the true reason for the retraction.

Which brings us back to the Stapel retractions. In the NYT article, which is well worth reading in its entirety, Stapel reveals how he got away with his bogus research for so long:

The key to why Stapel got away with his fabrications for so long lies in his keen understanding of the sociology of his field. “I didn’t do strange stuff, I never said let’s do an experiment to show that the earth is flat,” he said. “I always checked — this may be by a cunning manipulative mind — that the experiment was reasonable, that it followed from the research that had come before, that it was just this extra step that everybody was waiting for.” He always read the research literature extensively to generate his hypotheses. “So that it was believable and could be argued that this was the only logical thing you would find,” he said. “Everybody wants you to be novel and creative, but you also need to be truthful and likely. You need to be able to say that this is completely new and exciting, but it’s very likely given what we know so far.”

Science, itself, isn’t a guarantee of truth. Science is a method for finding the truth, if scientists are ruthlessly honest with other scientists, with the data, but most of all, with themselves. My belief is that the current practice of Science places far too little emphasis on replicating results with different scientists in different labs, using just the information provided in scientific papers. The proof of the pudding is in the eating, and the proof of research is in the replication. Simply finding new results isn’t enough, but the way we do Science today rewards the new and essentially ignores what happens next. The effects of dishonest research often is not confined to just the retracted paper. Scientists build on earlier results, and a bogus result can contaminate later research, sending honest scientists chasing down bunny trails. Being careful of unreplicated results is one way of dealing with this problem. Insisting peer-reviewed journals publish not only the paper but the raw data (and associated analysis tools!) used in the paper permits independent inspection of the reasoning in the paper. It turns out to be really hard to fake data! The ultimate technique for weeding out erroneous results lies in independent replication of claimed results, but that path is not valued today. We shouldn’t seek just the novel, but rather the true. The Scientific method has correctives for mistakes, inadvertent as well as dishonest, but we must take advantage of those techniques.

Richard Feynman said it best: Science is the belief in the ignorance of experts.