Experiments and Quantitative Research

By Janet Salmons, PhD Manager, Sage Research Methods Community

---

What is experimental research design?

Here is a brief overview from the SAGE Encyclopedia of Survey Research Methods:

Experimental design is one of several forms of scientific inquiry employed to identify the cause-and-effect relation between two or more variables and to assess the magnitude of the effect(s) produced. The independent variable is the experiment or treatment applied (e.g. a social policy measure, an educational reform, different incentive amounts and types) and the dependent variable is the condition (e.g. attitude, behavior) presumed to be influenced by the treatment. In the course of the experiment it is necessary to demonstrate the existence of covariation between variables, its nonspuriousness, and to show that the cause occurred before the effect. This sort of inquiry can take the form of an artificial experiment, carried out in a laboratory scenario, or a natural experiment implemented in a real-life context, where the level of control is lower. For both cases, the literature presents several taxonomies, from which four main types are considered: (1) true or classical experimental, (2) pre-experimental, single-subject experimental, and (3) quasi-experimental. In addition, there are a number of variations of the classic experimental design as well as of the quasi-experimental design.

In a true or classic experimental design, there are at least two groups of individuals or units of analysis: the experiment group and the control group. Participants are randomly assigned to both groups. These two groups are identical except that one of them is exposed to the experiment or causal agent, and the other, the control group, is not. In many instances, a pretest and a posttest are administered to all individuals in the two groups; but the pretest is not a necessary aspect of the true experiment. If there is a significant difference between members of the two groups, it is inferred that there is a cause-and-effect link between that treatment and the outcome.

The pre-experimental design does not have a control group to be compared with the experiment group. There is a pretest and a posttest applied to the same participants. In a single-subject experimental design, there is only one participant, or a small number, that is analyzed over a period of time. In quasi-experimental designs, participants come from naturally assembled or pre-determined groups (e.g. a family, a school class, a professional category, or inhabitants of a neighborhood) and are not therefore assigned randomly to the control and treatment groups.

An experimental design has to fulfill several conditions. The variables must be measured with accuracy and precision, and the statistical test must be denned before starting the experiment. If necessary, it must be possible to repeat the experiment in order to confirm that the outcome is statistically significant and that no other factors, other than the independent variable) the researcher manipulates, are responsible for the outcome. In practice, however, not every experiment meets all these conditions. Internal and external validity can be affected by several factors. For example, internal validity (cause-effect relation) can be influenced by the length of the time between the pretest and the posttest, by changes in the measurement instruments, the influence of the pretest on subsequent behavior, and so on, but can be improved with matching procedures and by randomization. External validity can be enhanced by the use of a representative sample and by avoiding artificial experiment settings lacking mundane realism.

In contrast, nonexperimental designs examine social phenomena without direct manipulation of the conditions that subjects experience. Subjects are not randomly assigned to different groups. As such, evidence that supports the cause-and-effect relationships is largely limited. Learn more and find examples of nonexperimental research in this post.

Open Access Experimental Studies

Dai, Y., & Luqiu, L. (2020). Camouflaged propaganda: A survey experiment on political native advertising. Research & Politics. https://doi.org/10.1177/2053168020935250

Abstract. We examine a new form of propaganda, political native advertising, in which political actors, including foreign governments, buy space in independent media outlets to publish advertisements that are camouflaged as standard news stories. Those who engage in this form of propaganda hope to exploit the higher credibility of the hosting media site to enhance the persuasiveness of their message. Despite the obvious political implications and ethical issues at stake, political native advertising has received almost no scholarly attention. Our article begins to redress this imbalance. Using an online survey experiment with real political native advertisements in the Washington Post and The Telegraph bought by the Chinese government, we provide some of the first empirical evidence on basic but important features of political native advertising. We find, among other things, that respondents struggle to distinguish political advertisements from standard news stories regardless of their level of education and media literacy, that political advertisements are more convincing if they appear on and are perceived as news from an independent hosting media site than in a government-controlled news outlet, and that trust in the hosting media site declines if the political advertisement is detected.

Gavriilidou, A., Wierbos, M. J., Daamen, W., Yuan, Y., Knoop, V. L., & Hoogendoorn, S. P. (2019). Large-Scale Bicycle Flow Experiment: Setup and Implementation. Transportation Research Record, 2673(5), 709–719. https://doi.org/10.1177/0361198119839974

Abstract. Cycling research at the operational behavioral level is limited, mainly because of the lack of empirical data. To overcome this data shortage, we performed a controlled, large-scale cycling experiment in the Netherlands. In this paper we describe the methodology for setting up and implementing such an experiment, from the motivation of its design using a conceptual model describing cyclist behavior to adjustments that were required during the experiment. The main contribution of this paper is, therefore, to be used as a guide in future experimental data collections. Moreover, we present the characteristics of the participants and their bicycles, and provide a qualitative description of phenomena observed during the experiment. Finally, we elaborate on the potential that the collected dataset holds for future research into understanding and modeling operational cycling behavior.

Kapelner, A., & Weinberg, D. (2019). Do Readers Judge Books by Author Gender? Results from a Randomized Experiment. Socius. https://doi.org/10.1177/2378023119868893

Abstract. We run a randomized experiment to examine gender discrimination in book purchasing with 2,544 subjects on Amazon’s Mechanical Turk. We manipulate author gender and book genre in a factorial design to study consumer preferences for male versus female versus androgynous authorship. Despite previous findings in the literature showing gender discrimination in book publishing and in evaluations of work, respondents expressed no gender preference across a variety of measures, including quality, interest, and the amount they were willing to pay to purchase the book. This nonfinding, if it holds up to additional research, suggests that book consumers may not express the same discriminatory tendencies observed among indie and traditional publishers.

Madsen, A. K., & Munk, A. K. (2019). Experiments with a data-public: Moving digital methods into critical proximity with political practice. Big Data & Society. https://doi.org/10.1177/2053951718825357

Abstract. Making publics visible through digital traces has recently generated interest by practitioners of public engagement and scholars within the field of digital methods. This paper presents an experiment in moving such methods into critical proximity with political practice and discusses how digital visualizations of topical debates become appropriated by actors and hardwired into existing ecologies of publics and politics. Through an experiment in rendering a specific data-public visible, it shows how the interplay between diverse conceptions of the public as well as the specific platforms and data invoked, resulted in a situated affordance-space that allowed specific renderings take shape, while disadvantaging others. Furthermore, it argues that several accepted tropes in the literatures of digital methods ended up being problematic guidelines in this space. Among these is the prescription to shown heterogeneity by pushing back at established media logics.

Matzkin, S., Shandler, R., & Canetti, D. (2023). The limits of cyberattacks in eroding political trust: A tripartite survey experiment. The British Journal of Politics and International Relations, 0(0). https://doi.org/10.1177/13691481231210383

Abstract. To what extent do security threats – such as cyberattacks – undermine trust in government? Fears have emerged that cyberattacks undercut public trust in government and sow doubt in democratic institutions heavily dependent on digitised systems. Nevertheless, the logic of this threat remains untested. This article presents survey experiments conducted in the United States (n = 607), the United Kingdom (n = 594), and Israel (n = 627) that examine whether cyberattacks on critical infrastructure truly undermine public trust in government and, if so, by what psychological mechanism. We exposed participants to simulations of cyberattacks against critical infrastructure before measuring the psychological and political outcomes. Our results reveal that cyberattacks do not undermine voters’ trust in the government’s ability to protect them. Furthermore, in the United States, exposure to cyberattacks heightens public trust by amplifying anger. Our findings inject a missing comparative component to the theoretical discussion of when, why, and how cyberattacks affect public trust in government.

Rong, K., Zhou, D., Shi, X., & Huang, W. (2022). Social Information Disclosure of Friends in Common in an E‐commerce Platform Ecosystem: An Online Experiment. Production and Operations Management, 31(3), 984-1005. https://doi.org/10.1111/poms.13591

Abstract. Real social information is believed to cause a trust–privacy paradox in e‐commerce platforms, with the degree of information disclosure affecting platform performance. In this study, we split the mixed influence into the direct trust mechanism and the indirect privacy mechanism, studying each in the context of partial and full social information disclosure, respectively. Using unique data from an 80‐day large‐scale online experiment conducted in an e‐commerce platform in China, we show that a partial social information disclosure could reduce the conversion rate, while a full social information disclosure could promote both the conversion and sales rates of the e‐commerce platform through a trust mechanism. Moreover, social information disclosure would reduce the sales rate and promote the conversion rate through a privacy mechanism. However, even though the degree of privacy could enhance the privacy mechanism, this mechanism is still much weaker than a trust mechanism. Overall, social information could play a positive role in e‐commerce platforms if it is fully disclosed. This study encourages cooperation between social media platforms and e‐commerce platforms by introducing a real social relationship into the online trading market.

Swanson, E., Kopotic, K., Zamarro, G., Mills, J. N., Greene, J. P., & W. Ritter, G. (2021). An Evaluation of the Educational Impact of College Campus Visits: A Randomized Experiment. AERA Open. https://doi.org/10.1177/2332858421989707

Abstract. We study whether visits to a college campus during eighth grade affect students’ interest in and preparation for college. Two cohorts of eighth graders were randomized within schools to a control condition, in which they received a college informational packet, or a treatment condition, in which they received the same information and visited a flagship university three times during an academic year. We estimate the effect of the visits on students’ college knowledge, postsecondary intentions, college preparatory behaviors, academic engagement, and ninth-grade course enrollment. Treated students exhibit higher levels of college knowledge, efficacy, and grit, as well as a higher likelihood of conversing with school personnel about college. Additionally, treated students are more likely to enroll in advanced science/social science courses. We find mixed evidence on whether the visits increased students’ diligence on classroom tasks and a negative impact on students’ desire to attend technical school.

van Gerwen, N., Buskens, V., & van der Lippe, T. (2018). Individual training and employees’ cooperative behavior: Evidence from a contextualized laboratory experiment. Rationality and Society, 30(4), 432–462. https://doi.org/10.1177/1043463118771428

Abstract. Employers are constantly seeking to improve employee performance by means of investing in employee training. The results of training are to a large extent dependent on employees’ willingness to behave productively in a cooperative manner. Yet, systematic evidence investigating the causal relation between training and employees’ cooperative behavior is rare. Here, we present results from a contextualized laboratory experiment in which subjects, who differ in terms of training participation, were asked to contribute resources to a team effort. We conclude that training promotes cooperative behavior, that is, voluntary contributions made to the team effort, in teams of employees working together for short periods of time. Training enhances cooperative behavior the most when provided to the higher skilled subjects. We also find that members in more stable teams act very cooperatively under all conditions, but their contributions do not increase further with training.

Books about Experimental Methods from Sage Publishing

Use the code COMMUNITY3 for a 20% discount when you order her book, valid worldwide until March 31, 2024.

Experimental Designs by Barak Ariel, Matthew Bland, Alex Sutherland (2022)

Get Your Data From Experiments by Helen Coleman (2021)

Experimental Evaluation Design for Program Improvement by Laura R. Peck (2019)

Designing Online Experiments for the Social Sciences by Giuseppe Veltri (2023). See an interview with Dr. Veltri to learn more.

---

Reference
Lavrakas, P. J. (2008). Encyclopedia of survey research methods (Vols. 1-0). Thousand Oaks, CA: Sage Publications, Inc. doi: 10.4135/9781412963947

---

More posts about quantitative research methods