On the hazards of significance testing. Part 2: the false discovery rate, or how not to make a fool of yourself with P values
Jump to follow-up
What follows is a simplified version of part of a paper that has now appeared as a preprint on arXiv. If you find anything wrong, or obscure, please email me. Be vicious: it will improve the eventual paper.
The paper has now appeared in the new Royal Society Open Science journal. There is a comments section at the end of the paper, for discussion. The first comment is from me, a correction of a typo that was spotted within hours. Luckily it’s pretty obvious.
It’s a follow-up to my very first paper, which was written in 1959 – 60, while I was a fourth year undergraduate (the history is ...
Source: DC's goodscience - March 24, 2014 Category: Science Authors: David Colquhoun Tags: false discovery rate statistics Bayesian P values significance Source Type: blogs
On the hazards of significance testing. Part 2: the false discovery rate, or how not to make a fool of yourself with P values
What follows is a simplified version of part of a paper that will shortly be submitted. If you find anything wrong, or obscure, please email me. Be vicious: it will improve the eventual paper.
It’s a follow-up to my very first paper, which was written in 1959 – 60, while I was a fourth year undergraduate.(the history is in a recent blog). I hope this one is better.
‘". . . before anything was known of Lydgate’s skill, the judgements on it had naturally been divided, depending on a sense of likelihood, situated perhaps in the pit of the stomach, or in the pineal gland, and differing in its...
Source: DC's goodscience - March 24, 2014 Category: Professors and Educators Authors: David Colquhoun Tags: false discovery rate statistics Bayesian P values significance Source Type: blogs
On the hazards of significance testing. Part 1: the screening problem
Jump to follow-up
This post is about why screening healthy people is generally a bad idea. It is the first in a series of posts on the hazards of statistics.
There is nothing new about it: Graeme Archer recently wrote a similar piece in his Telegraph blog. But the problems are consistently ignored by people who suggest screening tests, and by journals that promote their work. It seems that it can’t be said often enough.
The reason is that most screening tests give a large number of false positives. If your test comes out positive, your chance of actually having the disease is almost always quite small. False positiv...
Source: DC's goodscience - March 10, 2014 Category: Science Authors: David Colquhoun Tags: Screening statistics false positives Source Type: blogs
On the hazards of significance testing. Part 1: the screening problem
This post is about why screening healthy people is generally a bad idea. It is the first in a series of posts on the hazards of statistics.
There is nothing new about it, but the problems are consistently ignored by people who suggest screening tests, and by journals that promote their work. It seems that it can’t be said often enough.
The reason is that most screening tests give a large number of false positives. If your test comes out positive, your chance of actually having the disease is almost always quite small. False positive tests cause alarm, and they may do real harm, when they lead to unnecessary surgery ...
Source: DC's goodscience - March 10, 2014 Category: Professors and Educators Authors: David Colquhoun Tags: Screening statistics false positives Source Type: blogs
Some pharmacological history: an exam from 1959
Last year, I was sent my answer paper for one of my final exams, taken in 1959. This has triggered a bout of shamelessly autobiographical nostalgia.
The answer sheets that I wrote had been kept by one of my teachers at Leeds, Dr George Mogey. After he died in 2003, aged 86, his widow, Audrey, found them and sent them to me. And after a hunt through the junk piled high in my office, I found the exam papers from that year too.
George Mogey was an excellent teacher and a kind man. He gave most of the lectures to medical students, which we, as pharmacy/pharmacology students attended. His lectures were inspirational.
...
Source: DC's goodscience - February 6, 2014 Category: Science Authors: David Colquhoun Tags: B.L. Welch George Mogey H.O. Schild Pharmacology statistics University of Leeds inference J.W. Trevan UCL University College London Source Type: blogs
Some pharmacological history: an exam from 1959
Last year, I was sent my answer paper for one of my final exams, taken in 1959. This has triggered a bout of shamelessly autobiographical nostalgia.
The answer sheets that I wrote had been kept by one of my teachers at Leeds, Dr George Mogey. After he died in 2003, aged 86, his widow, Audrey, found them and sent them to me. And after a hunt through the junk piled high in my office, I found the exam papers from that year too.
George Mogey was an excellent teacher and a kind man. He gave most of the lectures to medical students, which we, as pharmacy/pharmacology students attended. His lectures were inspirational.
...
Source: DC's goodscience - February 6, 2014 Category: Professors and Educators Authors: David Colquhoun Tags: B.L. Welch George Mogey H.O. Schild Pharmacology statistics University of Leeds inference J.W. Trevan UCL University College London Source Type: blogs
La Trobe University (Melbourne) takes money to promote quackery
Jump to follow-up
This post is the original version of a post by Michael Vagg. It was posted at the Conversation but taken down within hours, on legal advice. Sadly, the Conversation has a track record for pusillanimous behaviour of this sort. It took minutes before the cached version reappeared on freezepage.com. I’m reposting it from there in the interests of free speech. La Trobe "university" should be ashamed that it’s prostituted itself for the sake of $15 m.
La Trobe’s deputy vice-chancellor, Keith Nugent, gives a make-believe response to the resignation of Ken Harvey in a video. It is, in...
Source: DC's goodscience - February 5, 2014 Category: Science Authors: David Colquhoun Tags: Academia corruption Keith Nugent La Trobe university Michael Vagg quackademia quackery Source Type: blogs
Why you should ignore altmetrics and other bibliometric nightmares
Conclusions about bibliometrics
Bibliometricians spend much time correlating one surrogate outcome with another, from which they learn little. What they don’t do is take the time to examine individual papers. Doing that makes it obvious that most metrics, and especially altmetrics, are indeed an ill-conceived and meretricious idea. Universities should know better than to subscribe to them.
Although altmetrics may be the silliest bibliometric idea yet, much this criticism applies equally to all such metrics. Even the most plausible metric, counting citations, is easily shown to be nonsense b...
Source: DC's goodscience - January 16, 2014 Category: Science Authors: David Colquhoun Tags: Academia altmetrics bibliometrics open access peer review Public relations publishing acupuncture badscience bibliobollocks publication regulation Source Type: blogs
Why you should ignore altmetrics and other bibliometric nightmares
Conclusions about bibliometrics
Bibliometricians spend much time correlating one surrogate outcome with another, from which they learn little. What they don’t do is take the time to examine individual papers. Doing that makes it obvious that most metrics, and especially altmetrics, are indeed an ill-conceived and meretricious idea. Universities should know better than to subscribe to them.
Although altmetrics may be the silliest bibliometric idea yet, much this criticism applies equally to all such metrics. Even the most plausible metric, counting citations, is easily shown to be nonsense b...
Source: DC's goodscience - January 16, 2014 Category: Professors and Educators Authors: David Colquhoun Tags: Academia altmetrics bibliometrics open access peer review Public relations publishing acupuncture badscience bibliobollocks publication regulation Source Type: blogs
Science is harmed by hype. How to live for 969 years.
Jump to follow-up
[This an update of a 2006 post on my old blog]
The New York Times (17 January 2006) published a beautiful spoof that illustrates only too clearly some of the bad practices that have developed in real science (as well as in quackery). It shows that competition, when taken to excess, leads to dishonesty.
More to the point, it shows that the public is well aware of the dishonesty that has resulted from the publish or perish culture, which has been inflicted on science by numbskull senior administrators (many of them scientists, or at least ex-scientists). Part of the blame must attach to "bibliometricia...
Source: DC's goodscience - December 31, 2013 Category: Professors and Educators Authors: David Colquhoun Tags: Alzheimer's Anti-science Bad journalism Big Pharma Universities Vitamin altmetrics bibliobollocks bibliometrics fraud good science misconduct Peter Lawrence politics Schekman vice-chancellors Vitamin E Vitamins Source Type: blogs
We know little about the effect of diet on health. That’s why so much is written about it
One of my scientific heroes is Bernard Katz. The closing words of his inaugural lecture, as professor of biophysics at UCL, hang on the wall of my office as a salutory reminder to refrain from talking about ‘how the brain works’. After speaking about his discoveries about synaptic transmission, he ended thus.
"My time is up and very glad I am, because I have been leading myself right up to a domain on which I should not dare to trespass, not even in an Inaugural Lecture. This domain contains the awkward problems of mind and matter about which so much has been talked and so little can be said, and hav...
Source: DC's goodscience - November 18, 2013 Category: Professors and Educators Authors: David Colquhoun Tags: badscience Bernard Katz nutribollocks nutrition nutritional therapy randomisation randomization RCT regulation Academia causality quackery Source Type: blogs
Yet another incompetent regulator. The General Pharmaceutical Council is criminally negligent
Conclusions
The main conclusion from all of this is that the General Pharmaceutical Council is almost criminally negligent. It continues to allow pharmacists, Anthony Pinkus among them, to endanger lives. It fails to apply its own declared principles. The members of its Council, and Duncan Rudkin (its chief executive and registrar), are not doing their job.
Individual pharmacists vary a lot, from the superb to those who believe in quackery. Some, perhaps many, are embarrassed by the fact that their employer compels them to sell rubbish. It’s too much to expect that they’ll endanger their mortgage payments by sp...
Source: DC's goodscience - November 4, 2013 Category: Professors and Educators Authors: David Colquhoun Tags: Ainsworths Alliance Boots Anthony Pinkus Avogadro CAM General Chiropractic Council General Pharmaceutical Council homeopathy Royal Pharmaceutical Society Ainsworth's malaria meningitis pertussis pharmacists Pharmacy regulator Source Type: blogs
One incompetent regulator, the Professional Standards Authority, approves another, the CNHC
Jump to follow-up
The consistent failure of ‘regulators’ to do their job has been a constant theme on this blog. There is a synopsis of dozens of them at Regulation of alternative medicine: why it doesn’t work, and never can. And it isn’t only quackery where this happens. The ineptitude (and extravagance) of the Quality Assurance Agency (QAA) was revealed starkly when the University of Wales’ accreditation of external degrees was revealed (by me and by BBC TV Wales, not by the QAA) to be so bad that the University had to shut down.
Here is another example that you couldn’t make up.
...
Source: DC's goodscience - October 13, 2013 Category: Professors and Educators Authors: David Colquhoun Tags: Anti-science antiscience badscience CNHC NOS Professional Standards Authority Quality assessment regulation alternative medicine CAM Complementary and Natural Healthcare Council Harry Cayton National Occupational Stardards Ofquac Source Type: blogs
A review of Do You Believe in Magic, by Paul Offit. And a fine piece of timidity from Nature Medicine
Despite the First Amendment in the US and a new Defamation Act in the UK, fear of legal threats continue to suppress the expression of honest scientific opinion.
I was asked by Nature Medicine (which is published in the USA) to write a review of Paul Offit’s new book. He’s something of a hero, so of course I agreed. The editor asked me to make some changes to the first draft, which I did. Then the editor concerned sent me this letter.
Thank you for the revised version of the book review.
The chief editor of the journal took a look at your piece, and he thought that it would be a good idea to run it past ...
Source: DC's goodscience - August 27, 2013 Category: Professors and Educators Authors: David Colquhoun Tags: badscience Bait and switch herbal medicine herbalism regulation supplements Academia alternative medicine DSHEA Orin Hatch Paul Offit Proxmire quackademia quackery Source Type: blogs
