TwitterPanic, NHST, Wizards, and the Cult of Significance Again

****Warning, some bad language used: don't read if you're offended by that sort of thing****I haven’t done a blog in a while, so I figured I ought to. Having joined Twitter a while back, I now find myself suffering from TwitterPanic™, which is an anxiety disorder (I fully anticipate to be part of DSM-V) characterised by a profound fear that people will unfollow you unless you keep posting things to remind them of why it’s great to follow you. In the past few weeks I have posted a video of a bat felating himself and a video of my cat stopping me writing my textbook. These might keep the animal ecologists happy, but most people probably follow me because they think I’m going to write interesting things about statistics, and not because they wanted to see a felating bat. Perhaps I’m wrong, and if so please tell me because I find it much easier to get ideas for things to put online that rhyme with stats (like bats and cats) than I do about stats itself.Anyway, I need to get over my TwitterPanic, so I’m writing a blog that’s actually about stats. A few blogs back I discussed whether I should buy the book ‘the Cult of Statistical Significance’. I did buy it, and read it. Well, when I say I read it, I started reading it, but if I’m honest I got a bit bored and stopped before the end. I’m the last person in the world who could ever criticise anyone for labouring points but I felt they did. To be fair to the authors I think the problem was more that they were essentially discussing things that I already knew, and it’s always difficult to keep focus when you’re not having ‘wow, I didn’t know that’ moments. I think if you’re a newbie to this debate then it’s an excellent book and easy to follow.In the book, the authors argue the case for abandoning null hypothesis significance testing, NHST (and I agree with most of what they say – see this), but they frame the whole debate a bit like a war between them (and people like them) and ‘the sizeless scientists’ (that’s the people who practice NHST). The ‘sizeless scientists’ are depicted (possibly not intentionally) like a bunch of stubborn, self-important, bearded, cape-wearing, fuckwitted, wizards who sit around in their wizardy rooms atop the tallest ivory tower in the kingdom of elephant tusks, hanging onto notions of significance testing for the sole purpose of annoying the authors with their fuckwizardry. I suspect the authors have had their research papers reviewed by these fuckwizards. I can empathise with the seeds of bile that experience might have sewn in the authors’ bellies, however, I wonder whether writing things like ‘perhaps they [the sizeless scientists] don’t know what a confidence interval is’  is the first step towards thinking that the blue material with stars on that you’ve just seen would look quite fetching as a hat.I don’t believe that people who have PhDs and do research are anything other than very clever people, and I think the vast majority  want to do the right thing when it comes to stats and data analysis (am I naïve here?). The tone of most of the emails I get suggest that people are very keen indeed not to mess up their stats. So, why is NHST so pervasive? I think we can look at a few sources:

1. Scientists in most disciplines are expected to be international experts in their discipline, which includes being theoretical leaders, research experts, and drivers of policy and practice. On top of this they’re also expected have a PhD in applied statistics. This situation is crazy really. So, people tend to think (not unreasonably) that what they were taught in university about statistics is probably still true. They don’t have too much time to update their knowledge. NHST is appealing because it’s a very recipe-book approach to things and recipes are easy to follow.
2. Some of the people above, will be given the task of teaching research methods/statistics to undergraduates/postgraduates. Your natural instinct is to teach what you know. If you were taught NHST, then that’s what you’ll teach. You might also be doing a course that forms part of a wider curriculum and that will affect what you teach. For example, I teach second year statistics, and by the time I get these students they have had a year of NHST, so it seems to me that it will be enormously confusing for them if I suddenly say ‘oh, all that stuff you were taught last year, well, I think it’s bollocks, learn this instead’. Instead, I weave in some arguments against NHST, but in a fairly low key way so that I don’t send half of the year into mass confusion and panic. Statistics is confusing enough for them without me undermining a year of their hard word.
3. Even if you wanted to remove NHST from your curriculum, you might be doing your students a great disservice because reviewers of research will likely be familiar with NHST and expect to see it. It might not be ideal that this is the case, but that is the world as we currently know it. When I write up research papers I would often love to abandon p-values but I know that if I do then I am effectively hand-carving a beautiful but knobbly stick, attaching it to my manuscript, and asking the editor if he or she would be so kind as to send the aforementioned stick to the reviewers so that they can beat my manuscript with it. If your students don’t know anything about NHST are you making their research careers more tricky to negotiate?
4. Textbooks. As I might have mentioned a few million times, I’m updating Discovering Statistics Using SPSS (DSUS as I like to call it). This book is centred around NHST, not because I’m particularly a fan of it, but because it’s what teachers and people who adopt the book expect to see in it. If they don’t see it, they will probably use a different book. I’m aware that this might come across as me completely whoring my principles to sell my book, and perhaps I am, but I also feel that you have to appreciate from where other people come. If you were taught NHST, that’s what you’ve done for 10 or 20 years, that’s what you teach because that what you genuinely believe is the right way to do things, then the last thing you need is a pompous little arse from Brighton telling you to change everything. It’s much better to have that pompous little arse try to stealth-brainwash you into change: Yes, each edition I feel that I can do a bit more to promote approaches other than NHST. Subvert from within and all that.

So, I think the cult of significance will change, but it will take time, and rather than seeing it as a war between rival factions, perhaps we should pretend it’s Christmas day, get out of the trenches, play a nice game of football/soccer, compliment each other on our pointy hats, and walk away with a better understanding of each other. It’d be nice if we didn’t go back to shooting each other on boxing day though.The APA guidelines of over 10 years ago and the increased use of meta-analysis have, I think, had a positive impact on practice. However, we’re still in a sort of hybrid wilderness where everyone does significance tests and, if you’re lucky, people report effect sizes too. I think perhaps one day NHST will be abandoned completely, but it will take time, and by the time it has we’ll probably have a found a reason why confidence intervals and effect sizes are as comedic as sticking a leech on your testicles to cure a headache.I’ve completely lost track of what the point of this blog was now. It started off that I was going to have a rant about one-tailed tests (I’ll save that for another day) because I thought that might ease my TwitterPanic. However, I got side tracked by thinking about the cult of significance book. I now feel a bit bad, because I might have been a bit critical of it and I don’t like it when people criticise my books so I probably shouldn’t criticise other’s. I stuck a sweet wizard hat related honeymoon picture in to hopefully soften the authors' attitude towards me in the unlikely event that they ever read this and decide to despise me. I then took some therapy for dealing with worrying too much about what other people think. It didn't work. Once I’d thought about that book I remembered that I’d wanted to tell anyone who might be interested that I thought the authors had been a bit harsh on people who use NHST. I think that side track was driven by a subconscious desire to use the word ‘fuckwizardry’, because it made me laugh when I thought of it and Sage will never let me put that in DSUS4. The end result is a blog about nothing, and that’s making my TwitterPanic worse …PS Definitions:Fuckwizard: someone who does some complicated/impressive task in a fuckwitted manner but with absolute confidence that they are doing it correctly.Fuckwizardry: doing a complicated or impressive task in a fuckwitted manner but with absolute confidence that you are doing it correctly