Of Elephants and Effect Sizes – Interview with Geoff Cumming

We all know these crucial moments while analysing our hard-earned data – the moment of truth – is there a star above the small p? Maybe even two? Can you write a nice and simple paper or do you have to bend your back to explain why people do not, surprisingly, behave the way you thought they would? It all depends on those little stars, below or above .05, significant or not, black or white.

This famous “Null-Hypothesis Significance Testing” (NHST)-approach is hugely appealing – after all, it gives us a cut-off, and we do not have to think much further, it simply decides (for us) if or if not the effect in question exists. It should not surprise us that life is rarely as simple as this. Indeed, contrary to what students are often taught, there exists quite some controversy about this “cook-book style” of statistics (see e.g. Dienes, 2008; Gigerenzer, 2004). In his book, “Understanding the new statistics”, Geoff Cumming explains why our usual approach to statistical thinking is hugely flawed (see also Ioannidis, 2005, and this post in the JEPS Bulletin) which in turn means that psychology as a science is in deep trouble.

Focusing on arbitrary p-values is problematic as it motivates us to torture the data until the stars appear: selective reporting, dropping certain items, or testing a few more participants (also called p-hacking; Simonsohn, Nelson, & Simmons, 2014). But even without that, p-values must not to be trusted. P-values “dance” – meaning that if you draw a new sample from your population and test your hypothesis again and again, given H1 is true, the value of p differs greatly: P is a highly unreliable measure and not to be trusted (Cumming, 2014).

Thus, Geoff Cumming proposes a “New Statistics” (also Cumming, 2013, 2014), a new way of thinking about statistics and psychological research, where reporting data focuses on effect sizes and confidence intervals (but see Morey, Rouder, Verhagen, & Wagenmakers, 2014). But the ideas of the New Statistics is not only about p-values: Cumming argues that we also need to change the way how we do research in the first place. For example, all details of a study (procedure, selection of participants, sample sizes, measures, and statistical analyses) should be specified before we see any results, ideally pre-registered, but we also need to report the finished study in full detail. Crucially, when reporting our results, we should think about future meta-analysis – Cumming argues that we need meta-analyses to build a cummulative science of psychology that includes all relevant results, and not just the significant ones, thus avoiding the file drawer effect.

Just recently, the journal Basic and Applied Social Psychology has announced to “ban p-values” from all their publications (Trafimow & Marks, 2015) – enough reason to ask Geoff Cummings a few questions about his work and psychology. Geoff Cummings is a emeritus professor of La Trobe University, Melbourne, and his main research focusses at statistical cognition: how do people understand (if they understand) statistical concepts.

I gather that you come from visual / spatial attention – how did you start to become interested in statistical cognition?

As a kid growing up in Melbourne I was fascinated by physics, especially nuclear physics. I was of the generation who believed that splitting (or fusing) the atom would give us abundant clean energy, and solve the world’s problems. How things change!

But, how could they make the teaching of physics so incredibly tedious?! Even first and second year physics at university was so rote, so mechanical–not thinking and imagination. So in my third year I took Psychology 1 as an extra subject, and was hooked. They expected us to think and write and propose solutions to problems! Science as process, not mere established fact and theory. I dropped physics, finished my degree in maths and stats, then went to Oxford for a DPhil in experimental psychology.

My thesis, with Anne Treisman, was on visual attention, but I also taught lots of stats. From the start I found NHST frustrating, and took ages trying to convince myself how it works. Then I tried numerous ways to explain it to others. Do things really need to be this complicated? I taught it for 40+ years and am still asking that same question.

As an experimental psychologist I also wondered why the tortuous logic underlying NHST had been so little studied. Statistics naturally struck me as being about communication–formulate a message, ideally with pictures, to convey a story to your readers. So it’s as much a question of perception and cognition as statistical models. Statistical cognition should be one of psychology’s gifts to all of science, by providing evidence to guide better statistical practices.

When and how did you realize that something was wrong with the way we conduct science?

I regarded it as a problem with the way we drew conclusions from data, rather than any broader issue. Only in the last few years have I seen how it’s so central. John Ioannidis did us a wonderful service by writing his 2005 article ‘Why most published research findings are false’, and explaining that over-reliance on .05 was the key underlying problem. It was so good the message came from medicine, not psychology–we might actually take it seriously.

In your paper as well as on your homepage you call for a change in attitude towards NHST. I was wondering how we can join the crusade. Where do you see the role of students?

Hey students–you are our future, you are our discipline, you are the big hope. Bottom line is that any scientist should strive for best research practice, whether in choosing measures, paradigms, techniques–or statistical methods. We’re all responsible for weighing the possibilities in all those aspects of our research, making our choices, and being prepared to explain and defend what we think is best.

OK, but back on planet Earth… I’m delighted to say it’s getting less difficult day by day. The APA Publication Manual recommends basing interpretation on point and interval estimates, Psychological Science has revised its guidelines, and others are making moves in similar directions. There are more resources available. Even so, it’s a challenge. But be strong; it’s worth persisting. Take a sentence or two, or a page or two, to explain and justify what you think is the best statistical approach in your particular situation, then go for it. If your name is on it, it should be what you believe to be best practice.

What do you consider to be the biggest challenge in psychological science in the next 10 years? 

The elephant (not to mention the blue whale, the dinosaur, and the echidna) in every room is climate change. Our children and grandchildren will ask us only one question about this and the previous decade–why did we do so little, when the writing is so devastatingly clear on the wall? We could say that the climate science is now relatively settled and the only real uncertainty is the psychology of why we are so easily persuaded en masse into torpor by vested interests and short term self-interest. It’s all about attitudes, decision making, and behaviour change–core business for psychology, surely. There’s tons of scope in our discipline for imaginative research that can make a difference–for our species.

Is there something you wish someone had told you at the beinning of your career? 

Find your passion, find good mentors. Everyone knows that, but most of us still insist on learning it for ourselves.

P.S. I’m also very interested in Bayesian techniques. They may become much more widely used as the tools and materials improve. I’m advocating estimation because at present it seems to me more achievable, with better materials available now, even if we still need more and better. Show me how Bayesian techniques can be made readily accessible to beginning students and I’ll be extremely interested. We also need serious statistical cognition investigation of Bayesian techniques and how they can best be taught.

References

Cumming, G. (2012). Understanding the new statistics: Effect sizes, confidence intervals, and meta-analysis. New York, NY: Routledge.

Cumming, G. (2014). The New Statistics : Why and How. Psychological Science, 25(1), 7–29. doi:10.1177/0956797613504966

Cumming, G. (2013). Replication and p intervals: p value predict the future only vaguely, but confidence intervals do much better. Perspectives on Psychological Science. doi:10.1111/j.1745-6924.2008.00079.x

Dienes, Z. (2008). Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Basingstoke, UK: Palgrave Macmillan.

Gigerenzer, G. (2004). Mindless statistics. The Journal of Socio-Economics33(5), 587-606. doi:10.1016/j.socec.2004.09.033

Ioannidis, J. P. a. (2005). Why most published research findings are false. PLoS Medicine, 2(8), e124. doi:10.1371/journal.pmed.0020124

Morey, R. D., Rouder, J. N., Verhagen, J., & Wagenmakers, E. J. (2014). Why Hypothesis Tests Are Essential for Psychological Science A Comment on Cumming (2014). Psychological science, 25(6), 1289-1290.

Simonsohn, U., Nelson, L. D., & Simmons, J. P. (2014). P-curve: A key to the file-drawer. Journal of Experimental Psychology: General, 143(2), 534–547. doi:10.1037/a0033242

Trafimow, D., & Marks, M. (2015). Editorial. Basic and Applied Social Psychology, 37(March), 1–2. doi:10.1080/01973533.2015.101299

 

 

Interview with Prof. Dermot Barnes-Holmes

 

Prof. Dermot Barnes-Holmes was a Foundation Professor at the Department of Psychology at National University of Ireland, Maynooth. He is known for his research in human language and cognition through the development of the Relational Frame Theory (RFT) with Steven C. Hayes, and its applications in various psychological settings. barnes_holmes_pic_edit

What I enjoy most about my job as a researcher … Supervising research students who are passionate about and genuinely interested in their research. Sharing what is often a voyage of intellectual discovery for both the student and me is still, after all these years, by far the most stimulating and enjoyable feature of what I do as an academic.

The biggest challenge in my career so far was … balancing being a new head of department (appointed in 1999) with a pivotal point in my research career (the writing and publication of the seminal text on Relational Frame Theory in 2001).

One research project I will never forget is…  There are at least three that really stand out for me. First, the writing of the 2001 RFT book with Steve Hayes and Bryan Roche, particularly the “crazy” multiple flying visits to Reno to work literally night and day with Steve on initial drafts of the book. Second, the research programme on the relating of derived relations, with Paul Smeets and Ian Stewart, as a model of analogical reasoning (from 1997-2005). And third, the last 10 years that have been devoted to the development of the Implicit Relational Assessment Procedure (IRAP), the Relational Elaboration and Coherence (REC) model, and the emergence from that work of the Multi-Dimensional Multi-Level (MDML) framework for analyzing arbitrarily applicable relational responding.

What I look for in a student who wants to work under my supervision … Simple – an individual who is motivated almost exclusively by the research and very little else.

Student research could be improved by … providing the supports necessary to encourage and sustain genuine creativity in the research process and minimize needless “box-ticking”.

Academically, I most admire … Steve Hayes …  because …. he has been like an intellectual father to me and, well, just look at the guy’s CV!

I wish someone had told me at the beginning of my career …  It’s okay, you’re on the right track – just sit back and enjoy the ride!

The largest changes in psychological science in the next 10 years will be … a far greater emphasis on how our research activity makes a meaningful difference in the real world (a change I welcome with open arms).

Interview with Prof. Alice Mado Proverbio

Prof. Alice Mado Proverbio has a degree in Experimental Psychology from the University of Rome “La Sapienza” and a PhD in General Psychology from the University of Padua. She did her Post Doctoral training at the University of California at Davis and at the University of Padua. As a research scientist at the University of Trieste, she guided the Cognitive Electrophysiology Laboratory from 1996 to 2000. Since 2001, she is Associate Professor of Psychobiology and Physiological Psychology at University of Milano-Bicocca. She founded the “Cognitive Electrophysiology” Lab at the same University in 2003. In 2014, she received the Habilitation as full Professor.

What I enjoy most about my job as a researcher …  Without a doubt what I enjoy most about my job as a researcher is the possibility to create and devise new experiments, to test new exciting ideas, to challenge pre-existing models with new hypotheses that I gather from discussions with people, but especially from a lot of reading and listening to insightful talks. It’s not rare that I get, what seems to be, a brilliant idea from reading or listening to scientists working outside my specific research field (cognitive electrophysiology). This can be genetics, evolutionary psychology, cellular biology, primatology or even molecular neuroscience. It can be something on Twitter, or even something that I spotted online. That’s what I like most: the creative process that precedes the actual experimental testing.

I also like that magic moment when, with my young co-workers standing all around my computer, we run the final ANOVA on a particular set of data we judge to be crucial to test our hypothesis. And we are all there, laughing and crossing our fingers, hoping for a high statistical significance, and then it gets p<0.005 and we all scream! I also love when an idea, just an incorporeal dream or a rough sketch at the beginning, but after months of working with my students, and refining details, re-adjusting the methodology, and changing the paradigm and all, finally becomes a consolidated paradigm, a concrete thing, almost a “person”, with a given personality and specific attitudes. We love to coin names for our new studies and paradigms, and stimulus types. Even computers and supplies and ERP components have personalized names in my lab. There are unofficial names (“just for us”) and more official, scientific terms that will be used later in the paper or in the dissertation.

The biggest challenge in my career so far was … there have been several challenging moments in my career, especially when I changed role, by becoming first a PhD student, then a Post-Doc fellow, a Researcher, and finally a Professor. Every passage required great effort in adjusting to the new situation and the many new commitments (not to mention, the new town or country, the new home, the new life, etc..). When I got a PhD student position I had to learn how to speak in public, deliver talks and travel a lot (while I enjoyed running experiments and writing my own papers). When I became a Post-Doc, I had to learn how to manage international relations and cooperate with multiple subjects and research groups. As a researcher, I had to face a lot of new work, mostly coming from student supervision, teaching and writing (books, chapters, papers), not to mention being the only person responsible for the ERP  lab. I often I had to work overnight. Becoming a Professor was very challenging at first, because of the large amount of teaching and lessons that I had to prepare for the first time. I learned how to be a good referee, a wise editor and the best mentor as possible for my students. I learned how to be very efficient with bureaucratic, administrative, and faculty duties, in order to have time for my research and my lab.

One research project I will never forget is…  I will never forget the research project aimed at testing the existence of possible subcortical inter-hemispheric pathways transferring visuomotor information in the brain of callosotomy (split-brain) patients, that I carried out in Ron Mangun’s lab in cooperation with Michel Gazzaniga, at the Center for Neuroscience of University of California at Davis. I had the extraordinary opportunity to test and get to know personally a beautiful person, the famous patient JW. I recall being incredibly excited and proud of my work at that time.

What I look for in a student who wants to work under my supervision … I mainly look for dedication, enthusiasm, patience, competence, rigor and loyalty, not necessarily in that specific order.

Student research could be improved by … I think that student research deserves the right equilibrium between autonomy and supervision. Sometimes I meet bright young researchers presenting poor pieces of evidence or lousy talks because of their inexperience mixed with a lack of supervision from their mentor. Its’ a real pity. Other times, I assist students acting as mere executors of projects they do not fully comprehend and testing hypotheses that they do not even scientifically understand. I think that students should not only perform the practical hands-on work in laboratories, but also do a lot of studying and reading to build a strong specialized knowledge.

Academically, I most admire … woman researchers (especially if independent and not grown under the wings of a powerful male mentor) …  because …. sometimes, they have to work twice as hard as their male colleagues, to prove their qualities. Indeed gender discrimination and inequalities of various types (from the most subtle to the most evident and gross inequalities) are still present at any level along the academic trail.

I wish someone had told me at the beginning of my career … I do not how to answer to this. I think that no advice can teach you better than your own personal experience. But I recall what I was actually being told, which revealed to be very useful in the hard times, and that is: do what you feel is better for you.

The largest changes in psychological science in the next 10 years will be … I am unsure what to predict. But I am pretty sure that the future is linked to a multidisciplinary integration, and that Psychology will grow only in interaction with other scientific disciplines, such as Cognitive Neuroscience, Genetics, Evolutionary Psychology, Cellular Neuroscience, Molecular Biology, Neuroimaging and the new emerging techniques (such as diffusor tensor imaging), and others that are still developing these days such as Brain Computer Interface (BCI), robotics.

Interview with Prof. Csikzentmihalyi

 

Prof. Mihaly Csikszentmihalyi is the Distinguished Professor of Psychology and Management at Claremont Graduate University and was the former head of the department of psychology at the University of Chicago. He is noted for his research on happiness and creativity, on which he published over 120 scientific articles and book chapters. He is also well known for introducing the concept of flow in his seminal work “Flow: The Psychology of Optimal Experience“. Csikszentmihalyi_Mihaly_WEB

What I enjoy most about my job as a researcher …  two things: the early analysis of data, when you are looking for patterns — exploring the psychological landscape, so to speak. Then the last part, when you start writing and trying to find the best way to express what you have learned.

The biggest challenge in my career so far was … to break out of the two reigning paradigms of my student’s days; the Freudian and the Skinnerian approaches.

One research project I will never forget is… perhaps the few months in 1968 when we started collecting data on the flow experience with a group of students at the college I was teaching at at the  time, Lake Forest College.

What I look for in a student who wants to work under my supervision … besides the obvious ones (academic and intellectual abilities): intrinsic motivation, a sense of humor, lack of excessive egotism.

Student research could be improved by … learning that what matters is engagement in a worth-while project.

Academically, I most admire … my friend Howard Gardner …  because …. he is an unselfish, sophisticated intellectual.

I wish someone had told me at the beginning of my career … how to get financial support for conducting large-scale research — although I probably would have ignored the advice anyway . . .

The largest changes in psychological science in the next 10 years will be … I am not a prophet, alas, so I have no idea. I know that the best-case scenario would be for psychology to focus on human experience, and establish conceptual links with other social sciences like sociology, anthropology, history, economics, and political science . . . The worst-case scenario would be selling out to neurobiology, and becoming a sub-discipline of that field. But I have no clue as to which of these two scenarios will win out in the evolutionary process.


Interview with Dr. Deirdre Barrett

Dr. Deirdre Barret is a researcher and lecturer at Harvard Medical School. She is well known for her research on dreams, hypnosis, and imagery. More recently she has written about evolutionary psychology and technology. She has also written severa successful books for the general public. deirdre barrett outside ucl 3a

What I enjoy most about my job as a researcher …  Any questions I have—in my case about dreams—I can come up with a way to operationalize the question and get an answer. Continue reading

Make the Most of Your Summer: Summer Schools in Europe


11051177_10205216017873360_1194271846_mWhy should you attend Summer Schools?

To put it simply: there is no better way to learn about psychology (and related disciplines), to travel, and to meet new people, all at the same time! Summer schools offer the opportunity to explore areas of psychology that might not be taught at your university, or to really explore a subject, seeing as this scheme allows you to  focus your work on one topic in the company of students who are enthusiastic about the same subject. Last year, I attended a summer school on Law, Criminology and Psychology – coming from Germany, where Criminology is in the Law faculty, that was my opportunity to learn more about eye-witness accounts, lie detection, psychopathy, and how to interrogate children. Aside from classic lectures, summer schools often include seminars and group work. Continue reading

Answering Frequently Asked Questions about JEPS

Is there anything you ever wanted to know about JEPS and the people behind it? Here are answers to our ten most frequently asked questions.

  1.  Who are we?

We are students from all over Europe and, as Editorial Team of the Journal of European Psychology Students (check out our Website here), we run JEPS.  Together with a group of other people (Associate Editors, Reviewers, Copyeditors, and Proofreaders), we see students’ manuscripts through the publication process.

Continue reading

Most frequent APA mistakes at a glance

APA-guidelines, don’t we all love them? As an example, take one simple black line used to separate words – the hyphen: not only do you have to check whether a term needs a hyphen or a blank space will suffice, you also have to think about the different types of hyphens (Em-dash, En-dash, minus, and hyphen). Yes, it is not that much fun. And at JEPS we often get the question: why do we even have to adhere to those guidelines?

APA_errors

Common APA Errors; Infographic taken from the EndNote Blog http://bit.ly/1uWDqnO

The answer is rather simple: The formatting constraints imposed by journals enable for the emphasis to be placed on the manuscript’s content during the review process. The fact that all manuscripts submitted share the same format allows for the Reviewers to concentrate on the content without being distracted by unfamiliar and irregular formatting and reporting styles.

The Publication Manual counts an impressive 286 pages and causes quite some confusion. In JEPS, we have counted the most frequent mistakes in manuscripts submitted to us – data that the EndNote-blog has translated into this nice little graphic.

Here you can find some suggestions on how to avoid these mistakes in the first place.

 References

American Psychological Association. (2009). Publication Manual of the American Psychological Association (6th ed.). Washington, DC: American Psychological Association.

Vainre, M. (2011). Common mistakes made in APA style. JEPS Bulletin, retrieved from http://blog.efpsa.org/2011/11/20/common-mistakes-made-in-apa-style/

A Psychologist’s Guide to Reading a Neuroimaging Paper

Psychological research is benefiting from advances in neuroimaging techniques. This has been achieved through the validation and falsification of established hypothesis in psychological science (Cacioppo, Berntson, & Nusbaum, 2008). It has also helped nurture links with neuroscience, leading to more comprehensive explanations of established theories. Positron Emission Tomography (PET), functional MRI (fMRI), structural MRI (sMRI), electroencephalography (EEG), diffusion tensor imaging (DTI) and numerous other lesser-known neuroimaging techniques can provide information complimentary to behavioural data (Wager, 2006). With these modalities of research becoming more prevalent, ranging from investigating the neural effects of mindfulness training to neuro-degeneration, it is worth taking a moment to highlight some points to help discern what may be good or poor research. Like any other methodology, neuroimaging is a great tool that can be used poorly. As with all areas of science, one must exercise a good degree of caution when reading neuroimaging papers. Continue reading

Interview in Israel: with Prof. Daniel Brom

Prof. Danny Brom is a clinical psychologist, the initiator of the Israel Trauma Coalition, and the Founding Director of the Israel Center for the Treatment of Psychotrauma in Jerusalem. Prof. Brom has published his first controlled outcome study on short-term therapy for Post Traumatic Stress Disorder in 1989, and has since published continuously on the topic. His main effort goes to bridging the gap between scientific data and service provision in the community.   2014-07-05 15.03.48

Continue reading