Good Science, Bad Science

A great bit of equine-related research has hit my desk lately: A group at Washington State University quantified lowered cortisol levels in kids enrolled in equine-facilitated learning programs. The study, a randomized trial, gives EFL programs a huge leg up in legitimization. It should help stakeholders and insurance companies understand what us horse owners already appreciate: being around horses is good for you. Read about it here.

The report is getting some decent traction on the Internet. No more attention, however, than some seriously sketchy science promoted by the International Society of Equitation Science (ISES).

ISES released a study by Cecilie Mejdell and her colleagues at the Norwegian University of Life Sciences that described how horses can choose for themselves whether or not to wear a blanket. The team trained 23 horses to read cue cards, essentially “asking” horses if they would like to wear a one.

I asked my vet about the research, she replied:

It sounds like a lot of assumptions were made – horses understanding consequences of their actions, for example.

• What if the preference was for the symbol rather than the action afterwards?
• How are they sure the message was communicated uniformly across the horses?
• What was the state of the horses’ body temperature under the conditions?
• What was the length/thickness of the hair coat at time of experience?
• Where did these horses’ come from and what sort of conditions were they exposed to prior to the study?

As a general rule, if a study begs more questions than it answers, something’s wrong.

My vet understood there’s a big difference between a proceedings journal (ISES) and published research (the EFL study). She also knows about the Gold Standard of Scientific Work and the Hierarchy of Evidence. Those principles state that meta analyses and randomized controlled trials are the best, most reliable and most respected kind of work.

The least reliable is expert opinion and case studies. ISES, though well-meaning in its efforts, promotes and disseminates mostly case studies. Read more about ISES.

That’s all well and good if you know how to review scientific studies. But what about the rest of us? Too often on the Internet, gold mixes with fool’s gold. Media outlets skim a study’s surface. If it’s fun and topical, they’ll give it good play. Readers on social media gobble it up without considering sources or fact checking.

The result? Lots of misinformation.

Want some tips on how to navigate through the junk to find the gems? Read tips for how to judge research here.

Read about getting caught reading junk here.