Moving your behavioral research online

By Jason Radford, CEO Volunteer Science and Research Program Manager at Northeastern University.

COVID-19 has affected research all over the world. With universities closing their campuses and governments issuing restrictions on social gatherings, behavioral research in the lab has ground to a halt. This situation is urgent. Ongoing studies have been disrupted and upcoming studies cannot begin until they are adapted to the new reality. At Volunteer Science, we’re helping researchers around the world navigate these changes. In this post, I’ll condense the most important recommendations we’re giving to researchers for translating their studies into an online format and recruiting virtual participants.

You can find a list of resources mentioned in this post here.

Building online studies

The first question we get is whether a study can be done online and what you need to do to move your study online. In general, nearly all behavioral studies can be done online. Many are still largely survey-based and already exist online through software like Qualtrics or SurveyMonkey. The studies done in cognitive psychology and economics that rely on in-lab software like z-Tree and E-Prime can now almost all be done online using general-purpose software like Volunteer Science or Gorila.sc or specialist software like jsPsych or oTree in psychology and economics, respectively.

However, there are some exceptions. Parts of a study that require special technical equipment like an FMRI machine typically cannot be done online. In some cases, the technical equipment can be shipped to participants. For example, biomarker data can still be collected through mail-in data collection kits. In other cases, the methods can be adjusted. In the case of eye tracking, cameras in computers and smartphones can still deliver eye-tracking data just at a lower quality than specialized devices. Finally, some parts of a study cannot be virtualized and so must be removed or replaced with a proxy method like a survey.

Once you know what parts of your study can be done online, the last question to answer is how you should build it. If your study involves standardized tools, you’re likely to find them in a library. In general, many software platforms including Volunteer Science have libraries of experiments you can use with relatively little programming. For example, the Experimental Factory has created a jsPsych-based library of over 100 tasks in cognitive psychology. If your study involves custom instruments, then you will have to build them yourself. Additionally, there are many packages for building online experiments from scratch in a variety of languages. PsychoPy, oTree, and Dallinger are based in python while Volunteer Science, Gorilla, and jsPsych use JavaScript. Some researchers use RShiny to do online experiments as well. Finally, some platforms like Gorilla.sc and oTree offer WYSIWYG editors for creating novel experiments.

Recruiting Participants

The other major part of the research plan that must be changed is participant recruitment. Many labs do not have their own virtual pool of participants from around campus and those that do may not be equipped to have subjects participate virtually. There are many options for recruiting subjects online and the needs vary substantially by field. Psychologists are often concerned with sample homogeneity while political scientists typically value heterogeneity and economists need to compensate people based on their performance. No one pool is the best for all research. Fortunately, there are many options.

First, there are paid subject pools like Mechanical Turk and Prolific. In these pools, you can pay subjects for their participation. The recommended rate of pay is the minimum wage equivalent. So, if minimum wage is $6 and the study takes ten minutes, researchers offer $1 per completion. While pay rates are not strictly enforced, academics typically pay above the minimum wage equivalent. The advantage to these pools is three-fold: responses come in quickly; you can typically recruit participants by nation, and at least some lifestyle and demographic features; and you can pay participants based on their performance which is necessary in economics. The disadvantage is that these samples may be demographically skewed, less naïve, and less attentive.

Second, companies like YouGov, Qualtrics, and Lucid sell access to existing online samples. You pay a flat rate and they send participants to your study. You typically pay more for selectivity. For example, it costs more for a subject from a randomized sample than a convenient one; or for a left-handed man aged between 25 and 40, than just any man. These samples can be expensive because they’re often of better quality with more and specific characteristics to target participants.

For those without money to spend on recruitment, you can recruit volunteers online using social media, your social networks, and crowd-science platforms. At Volunteer Science, we allow you to post your studies to our website and recruit whomever comes in. Other crowd-science platforms like SciStarter, Pavlovia, and Labvanced help researchers post their studies and recruit from the crowds they cultivate.

Whether you post your studies on one of these websites or host it yourself, you can tap into your social network and social media to reach the subjects you need to recruit. Websites like Reddit and Twitter can be great platforms for recruiting subjects if you can find the right way to reach your audience. Craigslist offers a volunteers section in each of its cities where you can post your call for participants regularly. You can also email friends, family, and faculty.

Lastly you can work with partner organizations willing to put your call for participants out to their members. This is especially for researchers who need a very specific population, say middle managers or firefighters. If your study is already online, most organizations have internal mailing lists they can use to spread the word on your behalf.

Now that researchers been doing behavioral studies online for over 30 years, there are many services and technologies available for you to do your study online. But it can be difficult to navigate all of these options if you’ve only dabbled in online research. I hope this blog has provided a map of the territory you can use to adapt your research plan.

### 

Volunteer Science: Build online studies and recruit online volunteers

Gorilla.sc: Build online studies using a WYSIWYG editor

Labvanced: Build online studies using a WYSIWYG editor

PsychoPy: Python package for creating psychology experiments

oTree: Python-based system for creating online economics experiments.

Dallinger: Python-based system for creating online experiments.

jsPsych: JavaScript-based system for creating online psychology experiments

RShiny: R-based system for creating scientific web applications

Mechanical Turk: Recruit paid online workers

Prolific: Recruit and pay highly-targeted study participants

YouGov: Recruit existing or new samples of participants

Qualtrics: Recruit existing or new samples of participants

Lucid: Recruit subjects from an aggregate of sample providers

SciStarter: Recruit volunteers interested in citizen science

Pavlovia: Recruit participants interested in psychology experiments

You can find more tools and technology for running online experiments and beyond here

About

Jason Radford is the CEO of Volunteer Science, visiting researcher at Northeastern University, and a PhD candidate in sociology at the University of Chicago. He develops computational methods for social research with a focus on networks, teams, and organizations.