BPS Hackathon – 21st June; LaTeX, R, Python goodness

Very exciting news here: I’ve just been invited to the first British Psychological Society (Maths, Statistics and Computing Section) Psychology open textbook hackathon!

Inspired by this event (where people got together and wrote an open-source maths textbook in a weekend) the day aims to raise awareness and skills, as well as perhaps produce some usable output.

The organisers are Thom Baguley of Nottingham Trent University (and the Serious Stats blog and book) and Sol Nte of Manchester University. They’ve very kindly invited me as a guest, so I’ll be hanging out and learning some new tricks myself, I’m sure.

Here’s the flyer for the event, with sign-up details etc. It’s free, but strictly limited to 20 places – if you’re keen, best be quick… (click the pic below for a bigger version):

 

Another miscellaneous grab-bag of goodies, links ‘n’ stuff

In lieu of a ‘proper’ post (forgive me, dear readers, the vicious task-masters at my proper job have been wielding the whip with particular alacrity recently) I’m putting together a list of links to cool things that I’ve come across lately.

So, in no particular order:

Tal Yarkoni’s outstanding Neurosynth website has now gone modular and open-source, meaning you can embed the code for the brain-image viewer into any website, and use it to present your own data – this is seriously cool. Check out his blog-post for the details.

An interesting little comment on “Why Google isn’t good enough for academic search”. Google scholar tends to be my first port of call these days, but the points made in this discussion are pretty much bang-on.

A fantastic PNAS paper by Kosinski et al. (2013; PDF) that demonstrates that personal attributes such as sexual orientation, ethnicity, religious and political views, some aspects of personality, intelligence and many others, can be automatically and accurately (to a fairly startling degree, actually) predicted merely from analysis of Facebook ‘Likes’. A fantastic result, that really demonstrates the value of doing research using online data.

Next up is Google Refine – an interesting little idea from Google intended to assist with cleaning up and re-formatting messy data. Looks like it could be promisingly useful.

A really seriously great website on the stats language R, designed to make the transition for SPSS and SAS users as easy as possible – very clear, very nicely explained. Beautiful stuff.

Another cool website called citethisforme.com; you fill in fields (author, title, etc.) for sources you wish to cite, and it creates a perfectly formatted bibliography for you in the style (APA, Harvard etc.) you choose. A cool idea, but in practice, filling out the fields would be incredibly tedious for anything more than a few sources. Good place to learn about how to format things for different types of reference though.

I’ve previously written about the use of U-HID boards for building USB response devices; I’ve just been made aware of a similar product called Labjack, which looks even more powerful and flexible. A Labjack package is included in the standard distribution of PsychoPy too, which is cool. I’m becoming more and more a fan of PsychoPy by the way – I’m now using it on a couple of projects, and it’s working very well indeed for me.

Now a trio of mobile apps to check out. Reference ME is available for both iOS and Android, and creates a citation in a specific style (Harvard, APA, etc.) when you scan the barcode of a book – very handy! The citations can then be emailed to you for pasting into essays or whatever.

The Great Brain Experiment is a free app from the Wellcome Trust (download links for both iOS and Android here) created in collaboration with UCL. The aim is to crowdsource a massive database on memory, impulsivity, risk-taking and other things. Give it a whirl – it’s free!

Lastly Codea is a very cool-looking iPad-only app that uses the Lua programming language to enable the (relatively) easy development and deployment of ‘proper’ code, entirely on the iPad. Very cool – Wired called it ‘the Garage Band of coding’, and while it’s probably not quite that easy to use, it’s definitely worth checking out if you want to use your iPad as a serious development tool.

If you’re still hungry for more internet goodies, I encourage you most heartily to check out my Links page, which is currently in an ongoing phase of rolling development (meaning, whenever I find something cool, I put it up there).

TTFN.

 

Why brain training is (probably) pernicious hogwash

The only treadmill your brain should be on is a hedonic one.

So-called brain-training tools seem to have exploded in the last few years; one estimate puts it at a $6 billion market by 2020. It’s clearly become a major industry, but what’s less clear is exactly what it does, and if it even works. The typical procedure seems to be to engage in short games, puzzles and working-memory-type tasks, and these are supposed to produce long term changes in attention, engagement and general fluid intelligence.

Whether this is actually true or not is a matter of some debate. I’m not a specialist in this area, but the received wisdom appears to be that training on specific tasks does improve performance – on those tasks. There seems to be little generalisation to other tasks, and even less to domain-general abilities like executive processing, or working memory. A high-profile study by Adrian Owen and colleagues (2010) reported exactly that – benefits in the tasks themselves, but little (if any) general benefits. A previous study from PNAS in 2008 does seem to contradict this, and reports an increase in fluid intelligence as a result of working-memory training – not only that, but they claim a dose-dependent effect, that is, more training = more increase in intelligence. The gains in that study were relatively small, and it should be also noted that the control group also apparently increased their intelligence somewhat over the same period as the experimental group – curious. There are lots of other studies around, but many have issues; small samples, poorly-controlled etc. etc.

So, the jury’s still very much out (though personally, I’m on the side of the skeptics on the issue). This hasn’t stopped a bewildering array of businesses starting up, making all kinds of wild claims, and playing on the fears of educators and parents that perhaps if they don’t provide these kinds of programs, their kids will be slipping behind the rest. All these companies have glossy, highly-polished, ethnically-balanced websites with testimonials, and lots of links to science-y looking videos that present their program as the only scientifically-proven method of increasing your child’s intelligence. A brief browse through some of these companies websites reveals that they range from the absurd (QDreams! Success at the speed of thought!) to the very, very slick indeed (e.g. Lumosity). Other examples are Cogmed (seems to be backed by Pearson publishers and, to its credit, links to a list of semi-relevant research papers), and the very simplistic PowerBrain Education - which seems to involve getting kids to do some odd-looking arm-shaking exercises. There’s literally hundreds of these companies. Some of them even seem to cater to businesses who want their employees to do these ‘exercises’.

LearningRX definitely falls into the slick category. According to this New York Times article it has 83 physical store-front franchises across the USA, where people can come to pay $80-90 an hour for one-on-one training, and they market this to parents as an alternative to traditional tutoring. A quick glance at their Scientific Advisory Board is pretty revealing – I count only one (clinical) psychologist, and a grab-bag of other professionals – mostly teachers (qualified to Masters level) with an optometrist, a chemical engineer and an audiologist. Not a single neuroscientist, and only a few qualified at doctorate level.

I’m not trying to be unnecessarily snobby about their qualifications here, I’m suggesting that the claims they make for their brain-training programs (literally: it will change your child’s life) are big ones, and we might expect that the people who developed it might be qualified in some area of brain-science. If it really, clearly worked, then of course it wouldn’t matter exactly who developed it, and what their qualifications were, but  there’s definitely reasonable doubt (if not outright disbelief) over its effectiveness.

And this is the important point. People are spending money on this - big money. Whether that’s a hard-pressed family struggling to find an extra $90 a week for their kid to have a session at one of LearningRX’s centres, or an education board deciding to institute one of these programs in its schools. Education budgets are tight enough, but these kinds of programs are being heavily invested in, and I can see why – they promise to make kids smarter, better-behaved, more attentive, and all you have to do is sit them in front of a special computer game for an hour a week. That must seem like a pretty attractive proposition for teachers. Unfortunately, if they really don’t work, then that money could be better spent on books, or musical instruments, or something else which might genuinely enrich the kids’ lives.

There’s a long and venerable history of unscrupulous people making money from pseudo-neuroscience – back in the 19th Century phrenology was described as “The science of picking someone’s pocket, through their skull.” I’d like to believe that some of these companies have a solid product that actually made a difference, but they all seem to have the whiff of snake-oil about them. For now I’m very much of the opinion that you’d probably be better off learning the piano, or Japanese, or even playing the latest Call of Duty. If you were really ambitious you could even try and get your kid to (Heaven forfend!) read the odd book now and again.

TTFN.

 

**Update 07/02/13**

I put that last sentence that mentions Call of Duty in there as a bit of flippancy, but I’ve since been informed (by Micah Allen on Twitter) of some evidence that playing action video games can indeed improve some cognitive processes such as the accuracy of visuo-spatial attention and reaction times. These results mostly originate from a single lab and so are in need of replication, but still – interesting. (I still reckon you’re probably better off with a good book though.)

Free, interactive MRI courses from Imaios.com (plus lots of other medical/anatomy material too)

A very quick post to point you towards a really fantastic set of online, interactive courses on MRI from a website called Imaios.com – a very nice, very slick set of material. The MRI courses are all free, but you’ll need to register to see the animations. Lots of other medical/anatomy-related courses on the site too – some free, some ‘premium’, and some nice looking mobile apps too.

Links page update

Just posted a fairly major update to my links page, including new sections on Neuropsychological/Cognitive testing, Neuromarketing/research businesses, and Academic conferences and organisations, plus lots of other links added to the existing sections, and occasional sprinkles of extra-bonus-added sarcasm throughout. Yay! Have fun people.

How to analyse reaction time (RT) data: Part 1

Reaction time tasks have been a mainstay of psychology since the technology to accurately time and record such responses became widely available in the 70s. RT tasks have been applied in a bewildering array of research areas and (when used properly) can provide information about memory, attention, emotion and even social behaviour.

This post will focus on the best way to handle such data, which is perhaps not as straightforward as might be assumed. Despite the title, I’m not really going to cover the actual analysis; there’s a lot of literature already out there about what particular statistical tests to use, and in any case, general advice of that kind is not much use as it depends largely on your experimental design. What I’m intending to focus on are the techniques the stats books don’t normally cover – data cleaning, formatting and transformation techniques which are essential to know about if you’re going to get the best out of your data-set.

For the purposes of this discussion I’ll use a simple made-up data-set, like this:

This table is formatted in the way that a lot of common psychology software (i.e. PsychoPy, Inquisit, E-Prime) records response data. From left-to-right, you can see we have three participants’ data here (1, 2, and 3 in column A), four trials for each subject (column B), two experimental conditions (C; presented in a random order), and then the actual reaction times (column D) and then a final column which codes whether the response was correct or not (1=correct, 0= error).

I created the data table using Microsoft Excel, and will do the processing with it too, however I really want to stress that Excel is definitely not the best way of doing this. It suits the present purpose because I’m doing this ‘by hand’ for the purposes of illustration. With a real data-set which might be thousands of lines long, these procedures would be much more easily accomplished by using the functions in your statistics weapon-of-choice (SPSS, R, Matlab, whatever). Needless to say, if you regularly have to deal with RT data it’s well worth putting the time into writing some general-purpose code which can be tweaked and re-used for subsequent data sets.

The procedures we’re going to follow with these data are:

1. Remove reaction times on error trials
2. Do some basic data-cleaning (removal of outlying data)
3. Re-format the data for analysis

1. Remove reaction times on error trials

As a general rule, reaction times from trials on which the participant made an error should not be used in subsequent analysis. The exceptions to this rule are some particular tasks where the error trials might be of particular interest (Go/No-Go tasks, and some others). Generally though, RTs from error trials are thought to be unreliable, since there’s an additional component process operating on error trials (i.e. whatever it was that produced the error). The easiest way of accomplishing this is to insert an additional column, and code all trials with errors as ‘0’, and all trials without an error as the original reaction time. This can be a simple IF/ELSE statement of the form:

IF (error=1) RT=RT,
ELSE RT=0

In this excel-based illustration I entered the formula: =IF(E2=1, D2,0) in cell F2, and then copied it down the rest of the column to apply to all the subsequent rows. Here’s the new data sheet:

2. Data-cleaning – Removal of outlying data

The whole topic of removing outliers from reaction time data is a fairly involved one, and difficult to illustrate with the simple example I’m using here. However, It’s a very important procedure, and something I’m going to return to in a later post, using a ‘real’ data-set. From a theoretical perspective, it’s usually desirable to remove both short and long outliers. Most people cannot push a button in response to, say, a visual stimulus in less than about 300ms, so it can be safely assumed that short RTs of, say, less than 250ms were probably initiated before the stimulus; that is, they were anticipatory. Long outliers are somewhat trickier conceptually – some tasks that involve a lot of effortful cognitive processing before a response (say a task involving doing difficult arithmetic) might have reaction times of several seconds, or even longer. However, very broadly, the mean RT for most ‘simple’ tasks tends to be around 400-700ms; this means that RTs longer than say, 1000ms might reflect some other kind of process. For instance, it might reflect the fact that the participant was bored, became distracted, temporarily forgot which button to push, etc. For these reasons then, it’s generally thought to be desirable to remove outlying reaction times from further analysis.

One (fairly simple-minded, but definitely valid) approach to removing outliers then, is to simply remove all values that fall below 250ms, or above 1000ms. This is what I’ve done in the example data-sheet in columns G and H, using simple IF statements of a similar form used for removal of the error trials:

You can see that two short RTs and one long one have been removed and recoded as 0.

3.  Re-format the data for analysis

The structure that most psychology experimental systems use for their data logging (similar to the one we’ve been using as an illustration) is not really appropriate for direct import into standard stats packages like SPSS. SPSS requires that one row on the data sheet is used for each participant, whereas we have one row-per-trial. In order to get our data in the right format we first need to sort the data, first by subject (column A), and then by condition (column C). Doing this sort procedure ensures that we know which entries in the final column are which – the first two rows of each subject’s data are always condition 1, and the second two are always condition 2:

We can then restructure the data from the final column, like so:

I’ve done this ‘by hand’ in Excel by cutting-and-pasting the values for each subject into a new sheet and using the paste-special > transpose function, however this is a stupid way of doing it – the ‘restructure’ functions in SPSS can accomplish this kind of thing very nicely. So, our condition 1 values are now in columns B:C and condition 2 values are in columns D:E. All that remains to do now would be to calculate summary statistics (means, variance, standard deviations, whatever; taking care that our 0 values are coded as missing, and not included in the calculations) for each set of columns (i.e. each condition) and perform the inferential tests of your choice (in this case, with only two within-subject conditions, it would be a paired t-test).

Next time, I’ll use a set of real reaction time data and do these procedures (and others) using SPSS, in order to illustrate some more sophisticated ways of handling outliers than just the simple high and low cutoffs detailed above.

TTFN.

Why every student needs a Google account

This post might seem a trifle umm… politically insensitive after recent revelations in the UK about exactly how much corporation tax Google pays (answer – basically none), but I’ve been planning it for a while, and unlike Starbucks (which should be boycotted at all costs, because their coffee sucks) Google is a little harder to avoid, and actually provides a whole slew of incredibly worthwhile, and mostly free, services. One of the first things you should do when you start an undergraduate course at a college/university is sign up for a Google account. Here’s why:

1. Gmail
You’ve probably already got an email address, but if you’re not using Gmail then you need to switch. The interface is brilliantly usable and customisable, and you get a massive 10Gb of storage for all your mail – more than you’ll likely ever need. The most important benefit though, is Gmail’s ability to pull all your current and future email accounts together in one place. Gmail can be set up as a POP3 client (here’s how) meaning it can pull email in from several different accounts and present it all in one inbox. You’ve probably got an account already, you’ll definitely get an account on your university’s servers, and when you leave and either go on to postgraduate study (maybe at a different university) or get a job, you’ll almost certainly get given yet another account. Gmail can centralize everything, and mean that you only have to check one inbox for all your accounts. You can even configure it so that it sends mail through, say, your university account by default, so people you contact see your ‘official’ email address. I’ve currently got five email accounts configured to read through Gmail, and I honestly couldn’t manage without it. Additionally, if you start using Gmail from day one, all your contacts and mail are saved in your Gmail account, and won’t be lost when you complete your course and your university account inevitably gets cancelled/deleted. Another benefit of Gmail is its ease of use with various smartphone platforms. Android (obviously) and iOS devices are designed to sync up with Google accounts pretty much seamlessly.

So, set up a Gmail account, and assume it’ll be your email address for life. Be sensible. Don’t choose a username like sexyluvkitten69@gmail.com, or gangzta4life@gmail.com – choose something you’ll be happy to put on a CV when you leave college, i.e. something that pretty much consists of your name.

2. GDrive/Docs
In one sense, Google Drive is a simple online storage locker for any kinds of files you like, a bit like Dropbox, or any of the other similar services which have proliferated recently. You get 5Gb of free space, and it’s easy to set up file sharing for specific other users, or to make your files available for download to anyone you send a link to. In another sense, it’s a full-featured web-based alternative to Microsoft Office, with the ability to create/edit documents, spreadsheets or presentations online, collaborate on them simultaneously with other users, and download them in a variety of the usual formats. Use it for just backing important things up, or use the full ‘Docs’ features – it’s up to you.

One other incredibly powerful feature of Google docs are the forms tools. These can be used to create online forms – the best way I currently know of to create online questionnaires for research purposes. The data from the questionnaires all gets dumped into a google docs spreadsheet for easy analysis too – very cool. This page has some good tips.

3. Google Scholar
Google Scholar is pretty much my first port-of-call for literature searches these days, and is often the best way of looking up papers quickly and easily. Yes, for in-depth research on a particular topic then you still need to look at more specialised databases, but as a first-pass tool, it’s fantastic. You can use it without being logged in with a Google account, but if you’re a researcher, you can get a Google Scholar profile page – like this: Isaac Newton’s Google Scholar profile page (only an h-index of 33 Isaac? Better get your thumb out of your arse for the REF old boy). This is the best way to keep track of your publications and some simple citation metrics.

4. Google Calendar
Yes, you need to start using a calendar. Google calendar can pull together several calendars together into one, sync seamlessly with your ‘phone, and send you alerts and emails to make sure you never miss a tutorial or lecture again. Or at least, you never miss one because you just forgot about it.

5. Blogger
Blogger is owned by Google, so if you want to start a blog (and it’s something you should definitely think about), all you need to do is go to blogger and hit a few buttons – simples. That’s the easy bit – then you actually have to write something of course…

6. Google Sites
Probably the easiest way to create free websites – as for Blogger above, you can literally create a site with a few clicks. Lots of good free templates that you can use and customise.

7. Google+
Yes, I know you use Facebook, but Google+ is the future. Maybe. The video hangouts are cool, anyway.

8. Other things
Use your Google account to post videos to YouTube, save maps/locations/addresses in Google Maps, find like-minded weirdos who are into the same things as you on Google Groups, read RSS feeds using Google Reader, and oooh… lots of other things.

Honestly, the feature of Gmail should be inducement enough for everyone to sign up for a Google account, the rest is just a bonus. Get to it people – it’s never too late to switch.

TTFN.

***UPDATE***

Following a couple of comments (below, and on Twitter) I feel it necessary to qualify somewhat my effusive recommendation of Google. Use of Google services inevitably involves surrendering personal information and other data to Google, which is a large corporation, and despite these services being free at the point of use, it should always be remembered that the business of corporations is to deliver profits. Locking oneself into a corporate system should be considered carefully, no matter how ‘convenient’ it might be. This article from Gizmodo is worth a read, as is this blog post from a former Google employee.

Programming experiments using PsychoPy – first impressions

I wrote a tiny post about PsychoPy a little while ago and it’s something I’ve been meaning to come back to since then. I’ve recently been tasked with an interesting problem; I need an experimental task for a bunch of undergrads to use in a ‘field study’ – something that they can run on their personal laptops, and test people in naturalistic environments (i.e. the participants’ homes). The task is based on a recent paper (Rezlescu et al., 2012) in PLoS One, and involves presenting face stimuli that vary in facial characteristics associated with trustworthiness, in a ‘game’ where the participant  plays the role of an investor and has to decide how much money they would invest in each person’s business. I was actually given a version of the experiment programmed (by someone else) in Matlab using the Psychtoolbox system. However, using this version seemed impractical for a number of reasons; firstly Matlab licences are expensive and getting a licenced version of Matlab on every student’s computer would have blown the budget available. Secondly, in my (admittedly, limited) experience with Matlab and Psychtoolbox, I’ve always found it to be a little… sensitive. What I mean is that whenever I’ve tried to transfer a (working) program onto another computer, I’ve generally run into trouble. Either the timing goes to hell, or  a different version of Matlab/Psychtoolbox is needed, or (in the worst cases) the program just crashes and needs debugging all over again. I could foresee getting this Matlab code working well on every single students’ laptop would be fraught with issues – some of them might be using OS X, and some might be using various versions of Windows – this is definitely going to cause problems.*

Somewhat counterintuitively therefore, I decided that the easiest thing to do was start from scratch and re-create the experiment using something else entirely. Since PsychoPy is a) entirely free, b) cross-platform (meaning it should work on any OS), and c) something I’d been meaning to look at seriously for a while anyway, it seemed like a good idea to try it out.

I’m happy to report it’s generally worked out pretty well. Despite being a complete novice with PsychoPy, and indeed the Python programming language, I managed to knock something reasonably decent together within a few hours. At times it was frustrating, but that’s always the case when programming experiments (at least, it’s always the case for a pretty rubbish programmer like me, anyway).

So, there are two separate modules to PsychoPy – the ‘Builder’ and the ‘Coder’. Since I’m a complete novice with Python, I steered clear of the Coder view, and pretty much used the Builder, which is a really nice graphical interface where experiments can be built up from modules (or ‘routines’) and flow parameters (i.e. ‘loop through X number of trials’) can be added. Here’s a view of the Builder with the main components labelled (clicky for bigness):

At the bottom is the Flow panel, where you add new routines or loops into your program. The large main Routine panel shows a set of tabs (one for each of your routines) where the events that occur in each of the routines can be defined on a timeline-style layout. At the right is a panel containing a list of stimuli (pictures, videos, random-dot-kinematograms, gratings etc.) and response types (keyboard, mouse, rating scales) that can be added to the routines. Once a stimulus or response is added to a routine, a properties box pops up which allows you to modify basic  (e.g. position, size, and colour of text) and some advanced (through the ‘modify everything’ field in some of the dialog boxes) characteristics.

It seems like it would be perfectly possible to build some basic kinds of experiments (e.g. a Stroop task) through the builder without ever having to look at any Python code. However, one of the really powerful features of the Builder interface is the ability to insert custom code snippets (using the ‘code’ component). These can be set to execute at the beginning or end of the experiment, routine, or on every frame. This aspect of the Builder really extends its capabilities and makes it a much more flexible, general-purpose tool. Even though I’m not that familiar with Python syntax, I was fairly easily able to get some if/else functions incorporating random number generation that calculated the amount returned to the investor on a trial, and to use those variables to display post-trial feedback. Clearly a bit of familiarity with the basics of programming logic is important to use these functions though.

This brings me to the Coder view – at any point the ‘Compile Script’ button in the toolbar can be pushed, which opens up the Coder view and displays a script derived from the current Builder view. The experiment can then be run either from the Builder or the Coder. I have to admit, I didn’t quite understand the relationship between the two at first –  I was under the impression that these were two views of the same set of underlying data, and changes in either one would be reflected in the other (a bit like the dual-view mode of HTML editors like Dreamweaver) but it turns out that’s not the case, and in fact, once I thought about it, that would be very difficult to implement with a ‘proper’ compiled language like Python. So, a script can be generated from the Builder, and the experiment can then be run from that script, however, changes made to it can not be propagated back to the Builder view. This means that unless you’re a serious Python ninja, you’re probably going to be doing most of the work in the Builder view. The Coder view is really good for debugging and working out how things fit together though – Python is (rightly) regarded as one of the most easily human-readable languages and if you’ve got a bit of experience with almost any other language, you shouldn’t find it too much of a problem to work out what’s going on.

Another nice feature is the ability of the ‘loop’ functions to read in the data it needs for each repeat of the loop (e.g. condition codes, text to be presented, picture filenames, etc.) from a plain text (comma separated) file or Excel sheet. Column headers in the input file become variables in the program and can then be referenced from other components. Data is also saved by default in the same two file formats – .csv and .xls. Finally, the PsychoPy installation comes with a set of nine pre-built demo experiments which range from the basic (Stroop) to more advanced ones (BART) which involve a few custom code elements.

There’s a couple of features that it doesn’t have which I think would be really useful – in particular in the Builder view it would be great if individual components could be copied and pasted between different routines. I found myself adding in a number of text elements and it was a bit laborious to go through them all and change the font, size, position etc. on each one so they were all the same. Of course ‘proper’ programmers working in the Coder view would be able to copy/paste these things very easily…

So, I like PsychoPy; I really do. I liked it even more when I transferred my program (written on a MacBook Air running OS X 10.8) onto a creaky old Windows XP desktop and it ran absolutely perfectly, first time. Amazing! I’m having a little bit of trouble getting it running well on a Windows Vista laptop (the program runs slowly and has some odd-looking artefacts on some of the pictures) but I’m pretty sure that’s an issue with the drivers for the graphics card and can be relatively easily fixed. Of course, Vista sucks, that could be the reason too.

So, I’d recommend PsychoPy to pretty much anybody – the Builder view makes it easy for novices to get started, and the code components and Coder view means it should keep seasoned code-warriors happy too. Plus, the holy trinity of being totally free, open-source, and cross-platform are huge advantages. I will definitely be using it again in future projects, and recommending it to students who want to learn this kind of thing.

Happy experimenting! TTFN.

*I don’t mean to unduly knock Matlab and/or Psychtoolbox – they’re both fantastically powerful and useful for some applications.

How to pilot an experiment

I got a serious question for you: What the fuck are you doing? This is not shit for you to be messin’ with. Are you ready to hear something? I want you to see if this sounds familiar: any time you try a decent crime, you got fifty ways you’re gonna fuck up. If you think of twenty-five of them, then you’re a genius… and you ain’t no genius.
Body Heat (1981, Lawrence Kasdan)

To consult the statistician after an experiment is finished is often merely to ask him to conduct a post-mortem examination. He can perhaps say what the experiment died of.
R.A. Fisher (1938)

Don’t crash and burn your experiment.

Doing a pilot run of a new psychology experiment is vital. No matter how well you think you’ve designed and programmed your task, there are (almost) always things that you didn’t think of. Going ahead and spending a lot of time and effort collecting a set of data without running a proper pilot is (potentially) a recipe for disaster. Several times I’ve seen data-sets where there was some subtle issue with the data logging, or the counter-balancing, or something else, which meant that the results were, at best,  compromised, and at worst completely useless.

All of the resultant suffering, agony, and sobbing could have been avoided by running a pilot study in the right way. It’s not sufficient to run through the experimental program a couple of times; a comprehensive test of an experiment has to include a test of the analysis as well. This is particularly true of any experiment involving methods like fMRI/MEG/EEG where a poor design can lead to a data-set that’s essentially uninterpretable, or perhaps even un-analysable. You may think you’ve logged all the data variables you think you’ll need for the analysis, and your design is a work of art, but you can’t be absolutely sure unless you actually do a test of the analysis.

This might seem like over-kill, or a waste of effort, however, you’re going to have to design your analysis at some point anyway, so why not do it at the beginning? Analyse your pilot data in exactly the way you’re planning on analysing your main data, save the details (using SPSS syntax, R code, SPM batch jobs – or whatever you’re using) and when you have your ‘proper’ data set, all you’ll (in theory) have to do is plug it in to your existing analysis setup.

These are the steps I normally go through when getting a new experiment up and running. Not all will be appropriate for all experiments, your mileage may vary etc. etc.

1. Test the stimulus program. Run through it a couple of times yourself, and get a friend/colleague to do it once too, and ask for feedback. Make sure it looks like it’s doing what you think it should be doing.

2. Check the timing of the stimulus program. This is almost always essential for a fMRI experiment, but may not be desperately important for some kinds of behavioural studies. Run through it with a stopwatch (the stopwatch function on your ‘phone is probably accurate enough). If you’re doing any kind of experiment involving rapid presentation of stimuli (visual masking, RSVP paradigms) you’ll want to do some more extensive testing to make sure your stimuli are being presented in the way that you think – this might involve plugging a light-sensitive diode into an oscilloscope, sticking it to your monitor with a bit of blu-tack and measuring the waveforms produced by your stimuli. For fMRI experiments the timing is critical. Even though the Haemodynamic Response Function (HRF) is slow (and somewhat variable) you’re almost always fighting to pull enough signal out of the noise, so why introduce more? A cumulative error of only a few tens of milliseconds per trial can mean that your experiment is a few seconds out by the end of a 10 minute scan – this means that your model regressors will be way off – and your results will likely suck.*

3. Look at the behavioural data files. I don’t mean do the analysis (yet), I mean just look at the data. First make sure all the variables you want logged are actually there, then dump it into Excel and get busy with the sort function. For instance, if you have 40 trials and 20 stimuli (each presented twice) make sure that each one really is being presented twice, and not some of them once, and some of them three times; sorting by the stimulus ID should make it instantly clear what’s going on. Make sure the correct responses and any errors are being logged correctly. Make sure the counter-balancing is working correctly by sorting on appropriate variables.

4. Do the analysis. Really do it. You’re obviously not looking for any significant results from the data, you’re just trying to validate your analysis pipeline and make sure you have all the information you need to do the stats. For fMRI experiments – look at your design matrix to see that it makes sense and that you’re not getting warnings about non-orthogonality of the regressors from the software. For fMRI data using visual stimuli, you could look at some basic effects (i.e. all stimuli > baseline) to make sure you get activity in the visual cortex. Button-pushing responses should also be visible as activity in the motor cortex in a single subject too – these kinds of sanity checks can be a good indicator of data quality. If you really want to be punctilious, bang it through a quick ICA routine and see if you get a) component(s) that look stimulus-related, b) something that looks like the default-mode network, and c) any suspiciously nasty-looking noise components (a and b = good, c = bad, obviously).

5. After all that, the rest is easy. Collect your proper set of data, analyse it using the routines you developed in point 4. above, write it up, and then send it to Nature.

And that, ladeez and gennulmen, is how to do it. Doing a proper pilot can only save you time and stress in the long run, and you can go ahead with your experiment in the certain knowledge that you’ve done everything in your power to make sure your data is as good as it can possibly be. Of course, it still might be total and utter crap, but that’ll probably be your participants’ fault, not yours.

Happy piloting! TTFN.

*Making sure your responses are being logged with a reasonable level of accuracy is also pretty important for many experiments, although this is a little harder to objectively verify. Hopefully if you’re using some reasonably well-validated piece of software and decent response device you shouldn’t have too many problems.

Video tutorial on designing and running psychology experiments using PsychoPy

PsychoPy is something which I’ve been meaning to write something substantive on for a while. Briefly though, it’s a system for designing and running experiments, programmed in the Python language, with a nice GUI front-end to it. I’ve only flirted with it briefly, but the open-source and cross-platform nature of it makes it a very attractive package for programming experiments, in my opinion. If I was learning this stuff for the first time, it’s definitely the system I’d use.

The purpose of this post was just to publicise a YouTube video, put up by the creator of PsychoPy – Jon Peirce, of Nottingham University. The video is a great little starter-tutorial for PsychoPy and gently walks the viewer through creating a simple experiment – great stuff.

Happy experimenting! Here’s the vid: