BrainDesire, Online Experiments, Neuro-business and a Plea for Openness

I’ve recently had some correspondence with the CEO of a company called Brain and Science which runs a site called BrainDesire. The site purports to use…

“…the latest discoveries in brain science, and applies them to dating, relationships and love. Discover BrainDesire and find out which men or women have a real potential for dating and for a serious relationship with you.”

It does this by an online test, where you enter the name of the person you are romantically interested in and the name is then used in a relatively simple priming experiment where the name is used as the prime and the task is a lexical decision. The idea is that the name of Mr or Ms Right leads to higher arousal, and therefore a faster reaction time for the lexical decision. Once the test is complete, your results are then available for a modest fee.

My initial reaction to this was “pffftt… yeah, right”, and expressing this opinion on twitter was what first attracted the attention of the CEO of the company. We ended up having quite a technical discussion by e-mail about the scientific basis of the procedure BrainDesire uses and my assessment of it is now much better informed. There are several potential issues with the test which were immediately apparent.

Firstly, the scientific basis of the test. It’s largely based on a 2006 paper by Bianchi-Demicheli, Grafton and Ortigue which demonstrates the reaction time speeding effect when a beloved’s name is used as a prime. These authors are well respected scientists who generally do outstanding work and I have no reason to suppose that there is anything wrong with the paper (yes, I could get nit-picky about the methods, but I could do that with almost any paper). Incidentally, Scott Grafton and Stephanie Ortigue comprise the Scientific Advisory Board of the company. What I do have an issue with though, is that this result has never really been convincingly replicated. There’s another paper from 2007 which uses the same task in a fMRI scanner, and pretty-much replicates the result, however the effects are much weaker and the sample size is much smaller, plus only women were used in this experiment. Not totally convincing.

Second, I have reservations about the methods used. Running reaction time (RT) experiments and getting accurate data is difficult – I’ve written about some of the issues with stimuli and response timing before. Briefly, in order to reveal differences in RT between conditions (which can be very small – on the order of a few tens of milliseconds) specialised software and response hardware is usually required. The BrainDesire site essentially runs a RT experiment in a browser window, and running experiments like this means you relinquish any control over hardware, operating system, or any other characteristics of the system the software is running on. You might think this wouldn’t make much difference, but I’ve run the test on several different systems and browsers, and it seems to – on some systems the primes are presented very fast and are basically not (consciously) seen, which is what you want when doing a priming task; on other systems the primes are pretty clearly visible.

Third is an issue of signal-to-noise. RT data are inherently noisy and consist of multiple component processes. Usually a lot of averaging is required (across trials, and across subjects) in any RT test in order to achieve reliable results. I’m sceptical that reliable results could be obtained from a 5-minute test on a single subject, particularly with the added noise-burden of the inconsistent hardware/software noted above.

Lastly, I have some issue with the marketing strategy. The site is chock-full of references to ‘the brain’ and ‘brain science’ and I don’t see how a simple reaction time task measures anything much brain-related. This is a minor gripe, and I feel a bit curmudgeonly even making it – of course advertising/marketing people spin things and exaggerate, and we’ve all got to earn a living etc. etc. Of course there are laws about false advertising, but do they even apply to the internet? I’ve no idea. Anyway, if psychologists/neuroscientists got upset every time a marketing agency or the media misrepresented something related to their discipline then we’d spend all day, every day in a red-misted blood-lust, and really, life’s too short.

So, I put the first three points above to the CEO when he contacted me, and his response was considered, helpful and generous – up to a point. Regarding the replication issue, he informed me that the experiment had in actual fact been extensively replicated, with more than 1000 participants, but that the data were collected after the intellectual property rights to the method were acquired by the company, were therefore not part of an academic project, and will not be published. He also acknowledged the issues related to inconsistent hardware/software on the client side, and the related issue of signal-to-noise, and informed me that there are several aspects of the BrainDesire test designed to cope with these issues. These include a selection mechanism which seeks to optimise the task for the particular client’s platform/browser, and some error-checking algorithms which discount potentially unreliable data. I found in my testing of the system that when I was deliberately sloppy and slow with my responses, then a prompt to pay more attention did appear, and also on some occasions the system told me that no results were available on completing the test. I was also told that there were some sophisticated digital-filtering algorithms used in the system for pulling the signal out of the noise, but that these formed proprietary knowledge and couldn’t be specified precisely for fear of giving up a competitive advantage.

I very much appreciate the willingness of this company to engage in a dialogue with me on this, and I find the idea behind the task and the way in which it’s implemented extremely interesting. The task as it stands seems pretty sophisticated, and a great deal of thought and effort has clearly been put into it. However the default position of a scientist is that of a sceptic, and because BrainDesire won’t make their validation data and methods public, no-one can be sure that what their test is measuring is valid or reliable. The CEO stated that he thought that scientific journals wouldn’t be very interested in their data/methods as it was nothing particularly new – this may or may not be the case, personally I’d be interested as it sounds like they’ve used some interesting and innovative methods. However, in that case I would argue that their validation data and details of the methods should be made available on their own website. If your business is built on a scientific finding which is even slightly questionable, you need to be prepared to have your methods questioned by interested (and annoying) scientists like me. I would argue that the benefits of publishing details of your data and methods in terms of publicity, and external peer-review would probably outweigh the costs in losing any competitive advantage; something that pharmaceutical companies have been forced to acknowledge over the past 10 years. The intersection of science and business is always going to be a sticky one, with different and perhaps conflicting philosophies rubbing up against each other, however, I would propose that if a key part of your business uses a (purportedly) scientific method, you have to be prepared for criticism on the same terms.

I think it likely that we will see more and more of these ‘neuro-business’ ventures in the future, and while there’s nothing wrong with that (hey, we’ve all got a mortgage to pay*), these companies should be prepared to defend their work in a rigorous manner, preferably by publishing details of their methods in the peer-reviewed scientific literature. Without that, no-one can be sure that what these businesses are doing is even nominally reliable or valid. I know BrainDesire is basically just a bit of fun, and if someone can make a bit of money out of it, then the best of luck to them, but the principle is still important.

I’d encourage any interested readers to give the BrainDesire test a try and let me know what you think. Comments very welcome…. TTFN.

*If anyone wants me to be a scientific consultant on one of them, then sign me up. No, I’m kidding. Or am I? Yes, I’m kidding. Maybe.

Advertisements

About Matt Wall

I do brains. BRAINZZZZ.

Posted on January 5, 2012, in Commentary, Cool new tech, Experimental techniques, Internet, Programming, Software and tagged , , , , , , , , , , . Bookmark the permalink. Leave a comment.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: