Skip to content

Introduction to survey methodology

Module items

Assignment

Sample assignment

Reading


Groves, Robert M., Floyd J. Fowler, Mick Couper, James M. Lepkowski, Eleanor Singer, and Roger Tourangeau. 2011. “An Introduction to Survey Methodology.” Pp. 1–37 in Survey Methodology. Somerset: Wiley.

Learning outcomes

  1. Define the survey and identify its defining characteristics
  2. Identify the uses of surveys
  3. Distinguish the question from the response category, and explain what makes each one good
  4. Compare closed-ended and open-ended questions
  5. Distinguish cross-sectional from longitudinal designs
  6. Examine ongoing surveys

What is [[survey]]?

  • A survey is a systematic method for gathering information from a sample of entities for the purposes of constructing quantitative descriptors of the attributes of the larger population of which the entities are members.
    • Surveys are used to collect information from or about people to describe, compare, or explain their knowledge, feelings, values, and behavior.
  • We conduct surveys so that information derived from a relatively small number of people accurately represents a significantly larger population.
    • The strategy involved in a survey is that we collect the same information about all the people in a sample.

[[Characteristics of survey]]

  • Surveys have the following characteristics:
    1. Information is gathered primarily by asking people questions.
    2. Information is collected either by having interviewers ask questions and record answers or by having people read or hear questions and record their own answers.
    3. Information is collected from only a subset of the population to be described-a sample-rather than from all members.

[[The uses of surveys]]

  • Surveys are perceived as a reflection of opinions.
  • Politicians rely on surveys for popular guidance in mapping out campaign strategies.
  • Commercial enterprises use survey findings to formulate market strategies.
  • Television and radio programs are evaluated and scheduled largely in accordance with the results of consumer surveys.

  • Government programs designed to provide assistance to various communities often rely on the results of surveys to determine program effectiveness.

  • Are surveys reliable?

    • Those who conduct surveys realize that the way questions are worded affects results and, if they are dishonest, they can frame their questions in a way that advances their personal agendas (Best, 2021).
    • When evaluating a survey, ask,
      1. Was the sample truly representative? That is, did all members of the total population surveyed have an equal chance of being selected?
      2. Were the questions clear and unambiguous?
      3. In the case of a mailed survey, did a significant number fail to respond?
      4. If so, how might non-respondents differ from respondents?
      5. Also, do other surveys corroborate the survey’s findings?

[[Elements of surveys]]: Questions and response categories

  • A survey is built out of two elements that must be designed together:
    • The [[question wording|question]]: the wording that tells every respondent exactly what information is being asked for.
    • The [[response category]]: the list of answer choices offered to the respondent, sometimes called the response categories, response alternatives, or answer scale.
  • A good question has two properties:
    1. Every respondent understands the question the same way, and
    2. Every respondent has the same perception of what counts as an adequate answer.
      • The simplest way to give respondents the same perception of an adequate answer is to hand them the list of acceptable answers, which is the job of the response category.

Why the response categories matters

  • Consider: "When did you have pneumonia?"
    • "Five years ago", "While I was in the army", "When I was 32", "In 1987" are all reasonable answers to the question as posed.
    • They are all appropriate, but they are not comparable, and descriptive statistics require comparable answers.
  • Rewriting it as "How old were you when you had pneumonia?" fixes the problem by specifying the terms of the answer.
  • A response category should be:
    • Exhaustive: every respondent can find an answer that fits, and
    • Mutually exclusive: no respondent fits two answers at once.

[[Question types]]

  • Survey questions are classified into two groups by whether a response category is supplied:
    • [[Closed-ended questions]]: the respondent chooses one (or more) answers from a set of alternatives provided by the researcher.
    • [[Open-ended questions]]: no list is provided; the respondent answers in their own words.

  • Virtually the same question can be written in either form:

    • Form Example
      Open-ended What do you consider to be the most important problem facing our country today?
      Closed-ended Here is a list of problems that many people are concerned about. Which do you consider to be the most important problem facing our country today?

      ○ The economy and cost of living
      ○ Healthcare
      ○ Immigration
      ○ Crime and public safety
      ○ The environment and climate change

The appeal of [[open-ended questions]]

  • Open-ended questions do have genuine advantages:
    • They can yield answers the researcher never anticipated,
    • They may describe the real views of respondents more closely,
    • Respondents like the chance to answer in their own words, and being able only to pick from a list can be a frustrating experience, and
    • They suit cases where the list of possible answers is too long to present.

Why we rarely ask [[open-ended questions]] in surveys

  • Despite those advantages, closed-ended questions are usually a more satisfactory way of creating data. Fowler (2014) gives four reasons:
    1. Respondents answer more reliably when the response alternatives are given to them.
    2. Researchers interpret answers more reliably when the alternatives were given, instead of coding respondents' free text after the fact.
    3. Open answers scatter. Many people give relatively rare answers that are not analytically useful; a constrained set of options makes it likely that enough people give any particular answer for it to be analytically interesting.
    4. Recording is easier. Since most data collection is computer assisted, checking a provided answer is far easier than keying in a narrative answer.

In practice

  • Most survey items are closed-ended, and open-ended items are used sparingly:
    • In pilot or pretest work, to discover the categories that the final closed-ended version will offer,
    • For quantities with a natural numeric answer (age, number of children, income in dollars), and
    • As an occasional "Other, please specify" escape hatch attached to a closed-ended item.

[[Types of survey design]]

  • Surveys are classified by their time dimension: how many points in time data are collected from respondents.
    • On this criterion, survey research falls into (at least) two fundamental types:
      • [[Cross-sectional surveys]] and,
      • [[Longitudinal surveys]].

[[Cross-sectional surveys]]

  • A cross-sectional survey collects data from a sample at a single point in time.
  • It produces a "snapshot" of the population; the distribution of attitudes, behaviors, or characteristics as they stand at the moment of measurement.
    • The opinion poll is the archetype: a sample of voters is asked about their voting intention on a given week, and the result describes that week.
  • Comparison is achieved between groups, not across time.
    • Example: after an anti-smoking campaign, ask one sample about current smoking and about campaign awareness, then compare high- and low-awareness respondents (de Vaus, 2014).

Strengths and weaknesses

  • Strengths:
    • Fast,
    • Comparatively cheap,
    • Requires only one round of fieldwork, and
    • Suffers no participant drop-out over time.
  • Weaknesses:
    • Confounding: the compared groups may differ in ways other than the presumed cause (age, health, education), so rival explanations must be removed statistically rather than by design,
    • Ambiguous causal order: one measurement cannot show which variable came first (do campaigns reduce smoking, or do non-smokers simply notice campaigns more?), and
    • Cannot measure change: it detects association, never development over time.

Example: The National Crime Victimization Survey

  • Field Description
    Sponsor U.S. Bureau of Justice Statistics
    Collector U.S. Census Bureau
    Purpose Develop detailed information about the victims and consequences of crime
    Year Started 1973
    Sampling Frame U.S. households
    Sample Size About 41,800 households (78,600 persons)
    Mode of Administration Paper questionnaire for 70% of the interviews, both face-to-face and telephone interviews; computer assistance for 30% of the interviews
    Frequency Monthly data collection

  • A question from The National Crime Victimization Survey

    • I'm going to read some examples that will give you an idea of the kinds of crimes this study covers.
    • As I go through them, tell me if any of these happened to you in the last 6 months, that is since __ (MONTH) __ (DAY), 20 __ .
      • Was something belonging to YOU stolen, such as:
        • a) Things that you carry, like luggage, a wallet, purse, briefcase, or book
        • b) Clothing, jewelry, or calculator
        • c) Bicycle or sports equipment
        • d) Things in your home, like a TV, stereo, or tools
        • e) Things from a vehicle, such as a package, groceries, camera, or cassette tapes

[[Longitudinal surveys]]

  • A longitudinal survey collects data at two or more points in time; its defining purpose is to ascertain change over a period.
  • Three common forms:
    1. [[Repeated cross-sectional design]]: the same questionnaire is given to fresh samples at each time point.
      1. Cheap, but shows only aggregate (net) change and differences may reflect sampling error.
    2. [[Panel design|Fixed-sample panel design]]: the same individuals are followed across waves, so change is measured at the individual level.
    3. [[Cohort study]]: a special case of the panel that follows people who share a common event or starting point (a birth year, a graduation year).

Strengths and weaknesses

  • Strengths:
    • Measures change directly rather than inferring it,
    • Establishes temporal order, strengthening causal claims, and
    • Panels and cohorts capture who changed, not just how much the total shifted.
  • Weaknesses:
    • Attrition: respondents drop out wave after wave, and drop-out is rarely random,
    • Costly and slow, and
    • Retrospective shortcuts (asking once about two time points) invite memory error and telescoping: misplacing events in time.

Example: The National Survey on Drug Use and Health

  • Field Description
    Sponsor Substance Abuse and Mental Health Services Administration
    Collector RTI International
    Purpose Provide estimates of rates of use, number of users, and other measures related to illicit drug, alcohol, and tobacco use at the state and national level
    Year Started 1971
    Sampling Frame U.S. households
    Sample Size 141,487 housing units; 67,870 persons
    Mode of Administration Face-to-face interview in respondent's home, with portions completed by respondent alone
    Frequency Yearly data collection

  • A question from The National Survey on Drug Use and Health

    • On the one day you smoked cigarettes during the past 30 days, how many cigarettes did you smoke?
      1. Less than one cigarette
      2. 1 cigarette
      3. 2 to 5 cigarettes
      4. 6 to 15 cigarettes (about ½ pack)
      5. 16 to 25 cigarettes (about 1 pack)
      6. 26 to 35 cigarettes (about 1 ½ packs)
      7. More than 35 cigarettes (about 2 packs or more)