Evidence, Skepticism, and the Scientific Method

Judith Luber-Narod, a high-school science teacher at the Abby Kelley Foster Charter Public School in Worcester, Mass., has incorporated climate change into her environmental studies classes, even though she teaches in a somewhat conservative area.

“I hesitated a little bit talking about something controversial,” she said. “But then I thought, how can you teach the environment without talking about it?”

Her students, on the other hand, love topics some deem controversial, she said. She devised an experiment in which she set up two terrariums with thermometers and then increased the level of carbon dioxide, the main greenhouse gas, in one of them.

The students watched as that terrarium got several degrees hotter than the other.

“I say to them, ‘I’m here to show you the evidence,’ ” she said. “ ‘If you want to believe the evidence when we’re done, that’s up to you.’ ”

I’m still working on that Dark Energy post, but it is proving to be ‘interesting’ to write. In the meantime, I wanted to talk a little about the role of experiment and skepticism in Science. The quote above comes from a New York Times science article New Guidelines Call for Broad Changes in Science Education. I don’t mean to be hard on the teacher. I do mean to be a little hard on the author and editors. But mostly, I’d like to use this as a cautionary tale showing why Good Science is not easy to do.

So, what’s wrong with the little experiment designed to show students how the greenhouse gas carbon dioxide raises Earth’s temperature? Almost everything. In particular, it is a great example of how a little knowledge is a dangerous thing, and how the role of experiment is often misunderstood.

First, the greenhouse effect is not really how a greenhouse warms. The glass of a greenhouse will indeed absorb infrared radiation, reradiating some heat–which would otherwise escape to the outside–back into the greenhouse. But this effect is quite minor, and real greenhouses warm because the glass enclosure blocks convection, preventing hot air from rising and being replaced by cooler air flowing in to take its place. It is (relatively) easy to demonstrate this by replacing the glass panes of a greenhouse with panes made of rock salt. The rock salt is transparent to infrared radiation, and so does not stop radiative cooling. The salt panes do block the formation of convective air currents just as well as glass. A greenhouse with rock salt panes will warm like a glass greenhouse, so the real warming mechanism is the elimination of convective flow and not the reduction in radiative cooling.

Similarly, in almost all terrarium experiments like the ones mentioned above, the real warming mechanism at work is not the carbon dioxide keeping the infrared radiation from carrying off heat energy, but the carbon dioxide inhibiting the formation of convective currents of air. The carbon dioxide, being heavier than air, stays within the open top terrarium. It doesn’t get hot enough to rise over the rim of the terrarium and allow cooler outside air to flow in. This is (relatively) easy to demonstrate by using argon gas instead of carbon dioxide. Argon is heavier than air, and argon is transparent to infrared radiation (like the rock salt). A terrarium filled with argon gas will heat just as well as one filled with carbon dioxide. Ergo, the warming effect has very little to do with the carbon dioxide reducing radiative cooling of the objects in the terrarium.

So, what about “I’m here to show you the evidence. If you want to believe the evidence when we’re done, that’s up to you” that the teacher claims? The problem is the experimental result (the terrarium warming) has more than one explanation, and the experiment isn’t designed to eliminate effects other than greenhouse gas style radiative warming. Good science is really hard, because even if you see a predicted effect, it is necessary to rule out alternative explanations for the observed evidence. If your hypothesis predicts A, but evidence shows B, the hypothesis is wrong. But if the hypothesis predicts A and the evidence shows A, this doesn’t necessarily show the hypothesis is correct. Experiments must be designed to test all other explanations for A and rule them out before the evidence shows the hypothesis is correct.

Science requires skepticism. Science requires more than even a theory agreeing with the evidence. Sometimes, what you see isn’t quite what you (or your teacher) think it is. Don’t be hasty to agree with authority. Be skeptical.


James Lovelock: The evolution of an intelligent man.

Damp winters on the edge of Dartmoor were taking their toll, so in recent years he has overwintered in St Louis, his wife’s hometown in Missouri. The experience altered his attitude to the politics and economics of energy.

Read the whole thing. The infirmities of age, combined with wintering over in the US, have changed many of his ideas.

I liked this quote, too. “He says he largely dismantled his home laboratory 10 years ago when he ended his life as a practising scientist: “I have become a thinker since then. There is so much more to do. I think retirement means death.”

The Key to Science, a mini lecture by Richard Feynman

This lecture snippet is really, really good.

The test of theory is observation. Use the theory to predict what happens, and look to see if that happens. If it doesn’t, the theory is wrong. No matter who came up with the theory, no matter their qualifications and credentials, no matter how lovely the math, if theory disagrees with Nature, it is nothing more than pretty math. If the theory’s predictions are observed in Nature, the theory is Not Wrong. Science never proces a theory correct, it just provides a method for weeding out incorrect theories.

Watch the whole thing.

Spelke-Pinker debate: The Science of Gender and Science

PINKER: But that makes the wrong prediction: the harder the science, the greater the participation of women! We find exactly the opposite: it’s the most subjective fields within academia — the social sciences, the humanities, the helping professions — that have the greatest representation of women. This follows exactly from the choices that women express in what gives them satisfaction in life. But it goes in the opposite direction to the prediction you made about the role of objective criteria in bringing about gender equity. Surely it’s physics, and not, say, sociology, that has the more objective criteria for success.

There is a nice article on Dr. Elizabeth Spelker in the NYT. In it, I found a link to a debate in 2005 between Dr. Steven Pinker and Dr. Spelke, both at Harvard, which was triggered by the (in)famous remarks of Larry Summers, then president of Harvard, on women in Science.

The debate is very interesting. They do not really differ on the facts, but on the interpretations. There is a video, and copies of the slide presentations, plus the text of the discussion at the end. Watch, listen, read, and make up your own mind.

Ivy tower research doesn’t always hold up in the Real World

During a decade as head of global cancer research at Amgen, C. Glenn Begley identified 53 “landmark” publications — papers in top journals, from reputable labs — for his team to reproduce. Begley sought to double-check the findings before trying to build on them for drug development.

Result: 47 of the 53 could not be replicated. He described his findings in a commentary piece published on Wednesday in the journal Nature.

Seeing irreproducible or misleading results published as peer reviewed scientific results is extremely disheartening. Science absolutely depends upon honest disclosure of experimental results along with transparent discussion of any real or suspected problems. Scientific papers designed not to further understanding but simply to further careers ought to result in career destruction. This sort of  dishonest research work harms Science, harms society, and ultimately harms humanity. Richard Feynman said it best in his famous Cargo Cult Science commencement address at Caltech…

It’s a kind of scientific integrity, a principle of scientific thought that corresponds to a kind of utter honesty–a kind of leaning over backwards. For example, if you’re doing an experiment, you should report everything that you think might make it invalid–not only what you think is right about it: other causes that could possibly explain your results; and things you thought of that you’ve eliminated by some other experiment, and how they worked–to make sure the other fellow can tell they have been eliminated.

Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can–if you know anything at all wrong, or possibly wrong–to explain it. If you make a theory, for example, and advertise it, or put it out, then you must also put down all the facts that disagree with it, as well as those that agree with it. There is also a more subtle problem. When you have put a lot of ideas together to make an elaborate theory, you want to make sure, when explaining what it fits, that those things it fits are not just the things that gave you the idea for the theory; but that the finished theory makes something else come out right, in addition.

Why Skepticism in Science isn’t just Politics

The reasons for the intense scepticism about OPERA are both general and specific.  The general reasons stem from the track record of experiments on the frontiers of science, which is pretty dismal.  This is not because experimentalists are careless or foolhardy (well, occasionally this happens) but because doing first-of-a-kind experiments, using new and clever methods and the latest technology, is extremely difficult, and prone to unforeseen problems.  And statistical flukes can always happen, too.  Everyone who has worked in high-energy physics for a while knows that the vast majority of exciting results, even from the best experimentalists, simply don’t hold up over time.  I made an informal list over the weekend of false alarms that have occurred during the nearly 30 years that I’ve been following or actually doing high-energy physics, and came up with nearly two dozen separate incidents — and I keep thinking of new ones.  [I may do some writing later this week about how some of these “discoveries” went awry.]  Meanwhile I can think of only three actual discoveries that survived, one of which (the top quark) was expected, one of which (neutrino oscillations) was pretty exciting but not unexpected, and only one of which really violated the prejudices of my field.  The last — the only real shocker to occur during my career — won this year’s Nobel Prize: the discovery that the universe’s expansion is accelerating instead of decelerating.

First, read the whole article by Dr. Strassler. I’ll wait.

OK. Prof. Strassler is exactly correct; whenever interesting results come out in physics, it pays to be skeptical. This isn’t because physicists want to protect the current paradigm, but a response born of long experience. Most interesting results have a good chance of being wrong. Nature always has the last say, and if an interesting result can be replicated, well, everyone wants to be part of a physics revolution. But if a result cannot be reproduced… it doesn’t matter how beautiful the math or how much explanatory power a theory has, at the end of the day, we can only accept those explanations that match up with the behavior of Nature. Feynman said it best: “It doesn’t matter how beautiful your theory is or how smart you are, if it doesn’t agree with experiment, it is wrong.”

Natural science has this wonderful property that an objective standard exists for judging the correctness of explanations. The behavior of Nature cannot be dismissed.

LHC results put supersymmetry theory ‘on the spot’

“It’s a beautiful idea. It explains dark matter, it explains the Higgs boson, it explains some aspects of cosmology; but that doesn’t mean it’s right.

“It could be that this whole framework has some fundamental flaws and we have to start over again and figure out a new direction,”


This is how Science is supposed to work. Supersymmetry is a beautiful theory with lots of explanatory power. But if the predictions don’t match with observational evidence, well, it is just pretty math.

Supersymmetry hasn’t been completely ruled out, yet, but the versions that remain viable are more theoretically complex, and that is never good news. Nature usually sides with elegant ideas. Sadly, one of the elegant ideas to lose out will be String Theory, which requires Supersymmetry.

What Is Science?

When someone says, “Science teaches such and such,” he is using the word incorrectly. Science doesn’t teach anything; experience teaches it. If they say to you, “Science has shown such and such,” you might ask, “How does science show it? How did the scientists find out? How? What? Where?”

It should not be “science has shown” but “this experiment, this effect, has shown.” And you have as much right as anyone else, upon hearing about the experiments–but be patient and listen to all the evidence–to judge whether a sensible conclusion has been arrived at.

Richard Feynman in a speech to the National Science Teachers Association in 1966.

Agnotology, Agnoiology and Cognitronics

As Farhad Manjoo notes in True Enough: Learning to Live in a Post-Fact Society, if we argue about what a fact means, we’re having a debate. If we argue about what the facts are, it’s agnotological Armageddon, where reality dies screaming.

New words! Always nice to have some new words at hand.

I disagree with the Manjoo quote above. It is not always obvious what the facts are, and reasonable people can and do disagree about the fact-ness of a large number of claims about Objective Reality. So “if we argue about what the facts are,” it doesn’t seem to me to be all that horrible. For example, is it a fact that an increase in the cosmic ray flux results in an increase in cloud formation? If having a discussion about such a claim means “we argue about what the facts are,” well, bring on the agnotological Armageddon. All facts do not identify themselves by walking up and biting one’s rear. A large part of Everyday Science is a debate about “what the facts are.”

Once both sides agree that a claim is a fact, well, then “arguing about what the facts are” is silly. But getting to the point that we can have a debate about the meaning of facts? That’s nontrivial.

Quote of the week.

When I asked Oxburgh if [Keith] Briffa [CRU academic] could reproduce his own results, he said in lots of cases he couldn’t,” Stringer told us. “That just isn’t science. It’s literature. If somebody can’t reproduce their own results, and nobody else can, then what is that work doing in the scientific journals?

Real Science isn’t a candidate for the Journal of Irreproducible Results.