Perspectives on Survey Research Design
by Janet Salmons, PhD, Research Community Manager for SAGE Methodspace
Research design is the focus for the first quarter of 2023. Find the unfolding series here. Want to know more about survey research in particular? See these recent posts.
Open-Access Articles from SAGE Journals
These articles offer advice and recommendations for researchers who want to design studies using surveys to collect data.
Axinn, W. G., Wagner, J., Couper, M., & Crawford, S. (2021). Applying Responsive Survey Design to Small-Scale Surveys: Campus Surveys of Sexual Misconduct. Sociological Methods & Research, 0(0). https://doi.org/10.1177/00491241211031270
Responsive survey design is a technique aimed at improving the efficiency or quality of surveys by using incoming data from the field to make design changes. The technique was pioneered on large national surveys, but the tools can also be applied on the smaller-scale surveys most commonly used by sociologists. We demonstrate responsive survey design in a small-scale, list-based sample survey of students on the topic of sexual misconduct. We investigate the impact of individual incentive levels and a two-phase responsive design with changes to mode of contact as approaches for limiting the potential of nonresponse bias in data from such surveys. Our analyses demonstrate that a two-phase design introducing telephone and face-to-face reminders to complete the survey can produce stronger change in response rates and characteristics of those who respond than higher incentive levels. These findings offer tools for sociologists designing smaller-scale surveys of special populations or sensitive topics.
Grubert, E. (2019). Every Door Direct Mail in US survey research: An anonymous census approach to mail survey sampling. Methodological Innovations, 12(2). https://doi.org/10.1177/2059799119862104
Mail surveys remain a popular method of eliciting attitudinal information, but declining response rates motivate inquiry into new, lower cost methods of contacting potential respondents. This work presents methodological findings from a medium-sized (~12,000 addresses) mail survey testing a United States Postal Service direct mail product called Every Door Direct Mail as a low-cost approach to anonymous mail survey distribution. The results suggest that under certain conditions, Every Door Direct Mail can be a useful approach for mail survey distribution, with response rates similar to those observed with analogous first-class mailing approaches but lower cost per response. As a tool for postal carrier-route saturation mailing that does not use names or addresses, Every Door Direct Mail is potentially useful for researchers who work in small, specific geographies or value or require anonymity. The results from this work suggest good performance on demographics and socially undesirable answers for Every Door Direct Mail relative to addressed mailings. The major disadvantages include an inability to conduct household-level probability sampling, an inability to customize nonresponse follow-up, and minimum mailing sizes associated with the postal carrier route saturation requirement. Every Door Direct Mail is unlikely to become a major tool for survey researchers, but it could be useful in niche applications. This study introduces Every Door Direct Mail to the survey methodology literature and presents empirical data intended to help researchers considering using Every Door Direct Mail.
Magliozzi, D., Saperstein, A., & Westbrook, L. (2016). Scaling Up: Representing Gender Diversity in Survey Research. Socius, 2. https://doi.org/10.1177/2378023116664352
Survey measures of gender have been critiqued for failing to reflect the diversity of the population. Conventionally, respondents to national surveys are categorized as female or male. Calls for improvement have centered on adding additional categories, such as transgender. We propose that in addition to revising categorical gender measures, national surveys should incorporate gradational measures of femininity and masculinity to better reflect gender diversity and sharpen models of gender inequality. Our results from two national pilot studies show that conventional measures mask significant variation within the categories of female and male. For example, less than a quarter of respondents reported that they are very feminine or masculine, respectively, and not at all the other. We also demonstrate that scale responses can be treated as independent variables in studies of inequality or as dependent variables that allow gender identification to be an outcome of social processes.
Mayer, A. (2021). Reducing respondents’ perceptions of bias in survey research. Methodological Innovations, 14(3). https://doi.org/10.1177/20597991211055952
Response rates for surveys have declined steadily over the last few decades. During this period, trust in science has also waned and conspiratorial theorizing around scientific issues has seemingly become more prevalent. In our prior work, we found that a significant portion of respondents will claim that a given survey is “biased.” In this follow-up research, we qualify these perceptions of bias and point to potential causes and ameliorative mechanisms.
O’Connor S. Designing and Using Surveys in Nursing Research: A Contemporary Discussion. Clinical Nursing Research. 2022;31(4):567-570. doi:10.1177/10547738211064739
Abstract. This commentary summarizes the contemporary design and use of surveys or questionnaires in nursing science, particularly in light of recent reporting guidelines to standardize and improve the quality of survey studies in healthcare research. The benefits, risks, and limitations of these types of data collection tools are also briefly discussed.
Scheitle, C. P. (2019). Does the public’s confidence in the scientific community affect its willingness to participate in social science research? A test examining the demeanor of survey respondents. Methodological Innovations, 12(3). https://doi.org/10.1177/2059799119884280
Research examining the consequences of the public’s confidence in the scientific community has primarily focused on the natural or medical sciences. It is not clear whether the public’s confidence in the scientific community has implications for research and practice in the social sciences. To begin examining this question, this study assesses whether survey respondents’ confidence in the scientific community is associated with their demeanor during the survey interview. This is consequential because respondent demeanor itself has been associated with survey refusal and nonresponse to items within surveys. Analysis of the 2004–2016 General Social Survey finds that individuals expressing more confidence in the scientific community are rated as having more positive demeanors by interviewers. Respondents’ confidence in other types of institutions does not show the same association, suggesting that confidence in the scientific community is uniquely associated with respondents’ demeanor during the interview. These findings suggest that the public’s confidence in science could have implications for at least survey-based social science research.
Van Quaquebeke, N., Salem, M., van Dijke, M., & Wenzel, R. (2022). Conducting organizational survey and experimental research online: From convenient to ambitious in study designs, recruiting, and data quality. Organizational Psychology Review, 12(3), 268–305. https://doi.org/10.1177/20413866221097571
Abstract. Conducting organizational research via online surveys and experiments offers a host of advantages over traditional forms of data collection when it comes to sampling for more advanced study designs, while also ensuring data quality. To draw attention to these advantages and encourage researchers to fully leverage them, the present paper is structured into two parts. First, along a structure of commonly used research designs, we showcase select organizational psychology (OP) and organizational behavior (OB) research and explain how the Internet makes it feasible to conduct research not only with larger and more representative samples, but also with more complex research designs than circumstances usually allow in offline settings. Subsequently, because online data collections often also come with some data quality concerns, in the second section, we synthesize the methodological literature to outline three improvement areas and several accompanying strategies for bolstering data quality.
Zhang, S., & Wagner, J. (2022). The Additional Effects of Adaptive Survey Design Beyond Post-Survey Adjustment: An Experimental Evaluation. Sociological Methods & Research, 0(0). https://doi.org/10.1177/00491241221099550
Adaptive survey design refers to using targeted procedures to recruit different sampled cases. This technique strives to reduce bias and variance of survey estimates by trying to recruit a larger and more balanced set of respondents. However, it is not well understood how adaptive design can improve data and survey estimates beyond the well-established post-survey adjustment. This paper reports the results of an experiment that evaluated the additional effect of adaptive design to post-survey adjustments. The experiment was conducted in the Detroit Metro Area Communities Study in 2021. We evaluated the adaptive design in five outcomes: 1) response rates, 2) demographic composition of respondents, 3) bias and variance of key survey estimates, 4) changes in significant results of regression models, and 5) costs. The most significant benefit of the adaptive design was its ability to generate more efficient survey estimates with smaller variances and smaller design effects.
Resources in SAGE Research Methods
SAGE Research Methods is a library database. This Reading List contains a collection of SAGE e-books, case studies, and videos, about survey research.
If you don’t have library access but would like to explore SAGE Research Methods, sign up for a free trial.
More Methodspace Posts about Research Design
Informed consent is the term given to the agreement between researcher and participant. In this post Janet Salmons offers suggestions about the intersections of the Internet communications, ethics and participants.