2008 AAPOR PRESENTATIONS
COMPUTING METRICS FOR ONLINE PANELS
Mario Callegaro, Knowledge Networks; Email
Charles DiSogra, Knowledge Networks; Email
There are many ways and different terminology used to compute response metrics for online panels. This paper discusses the sparse literature on how to compute response, attrition and re-interview rates and proposes formulas and a terminology that can be used to calculate these metrics for online panels. It starts from the distinction between probability-based panels and volunteer or opt-in panels. We then move to review references mostly from international organizations suggesting metrics for online panels. In order to propose response metrics we highlight the different recruitment steps in order to build an online panel. At each step non-response can creep in and it is important to describe recruitment mechanisms to properly understand non-response mechanisms. Complex panel design situations are considered in the last part of the paper. We conclude with a discussion of the meaning of the different metrics proposed and of the relationship between response rates and survey quality.
More information: this paper has been published in a Special Issue of Public Opinion Quarterly (Volume 72, Number 5, Spring 2009); to access this article, type this into your search engine: "Computing Metrics for Online Panels POQ".
Citation: Callegaro, Mario, and Charles DiSogra. 2008. Computing Response Metrics for Online Panels. Public Opinion Quarterly 72(5) pp. 1008-1031.
THE USE OF LOTTERY-TYPE-GAME ITEMS TO EXAMINE CHARITABLE GIVING AFTER KATRINA: A FIRST LOOK AT USING GAMES IN ONLINE PANEL RESEARCH
Christina Fong, Carnegie Mellon University, National Bureau of Economics
Poom Nukulkij, Knowledge Networks; Email
The use of online panel research creates unique opportunities in many different areas of research. Surveys administered using online panels are less susceptible to respondent biases (e.g., social desirability) that may be present in telephone or in-person surveys, potentially allowing for more accurate depiction of attitudes and behaviors. In this research, we present a study that takes advantage of the unique capabilities of online research. In June 2006, researchers affiliated with the National Bureau of Economic Research, Carnegie Mellon University and Knowledge Networks conducted a survey investigating peoples' attitudes and behaviors towards charitable donations to victims of Hurricane Katrina. Participants consisted of members of KnowledgePanel®, the RDD-recruited Internet panel managed by Knowledge Networks. Panelists were asked to allocate any portion of $100 to a charity in a state affected by Katrina or keep it for themselves; they could divide the amount in any way, using any criteria they saw fit. One critique of studies of this nature is that hypothetical behavior may not reflect actual behavior. To increase the salience of the experiment, participants were informed that Knowledge Networks would pay the dollar amounts indicated for one out of every ten participants. The "winners" who would have the money donated to charity or given to them were those whose randomly-selected number matched the first digit chosen in a pre-specified drawing from the Louisiana State Lottery Pick 3 lottery game. First, we will describe the study design, detailing the lottery game and context that was presented to participants. Information about the rules and regulations governing the use of lottery type games will also be presented. Next, we will provide a brief overview of demographic and panel characteristics related to donating behavior. Finally, we will discuss the fulfillment of these incentives with the selected charities and reactions from KnowledgePanel members.
More Info – PowerPoint | Full Paper
A NATIONAL EXPERIMENT EXAMINING THE EFFECTS OF POLITICAL ORIENTATION AND THE SALIENCE OF RACE AND LOCATION ON SELF-REPORTED CHARITABLE GIVING AFTER HURRICANE KATRINA
Linda J. Skitka, University of Illinois-Chicago
Elizabeth Mullen, University of Illinois-Chicago
Bill McCready, Knowledge Networks; Email
The purpose of this study was to experimentally examine the relationship between politically motivated reasoning, and the relative salience of race and location on self-reported charitable giving after Hurricane Katrina in a national random sample (N = 942). The experiment was a 2 (Locale: Gulfport, New Orleans) X 3 (Race Prime [using imbedded photographs]: None, White, Black) X 3 (Political Orientation: Democrat, Moderate, Republican) between-subjects experimental design with the dependent measure being self-reported recall of participants' charitable giving after Hurricane Katrina. Demographics and "personal experience with Katrina" were included as control variables. Results indicated that Republicans' self-reported charitable giving was affected by racial primes when they believed the images they saw represented damage to Gulfport, MS, but not when they believed the damage was to New Orleans. Republicans reported giving significantly more money when primed with white compared to black or no victims associated with Gulfport, but did not vary as a function of race primes associated with New Orleans. Democrats self-reported charitable giving was higher when photographic primes included pictures of people believed to be New Orleans hurricane victims than when not-primed with pictures of people, regardless of whether those picture primes were of Black or White victims of the catastrophe, but were unaffected by race in the Gulfport condition. Political moderates reported similar levels of charitable giving regardless of locale or race priming conditions. These results suggest that ideologues' memory or willingness to report money donated to charity are affected by subtle cues about race or context: Republicans reported giving more to whites when primed with a more stereotypically white and conservative context (Gulfport, MS), whereas Democrats reported giving more when primed with people rather than property damage in the more stereotypically black and Democrat context of New Orleans.
More Info – PowerPoint
LONGITUDINAL TRACKING OF VOTER PERCEPTIONS DURING THE 2007-2008 ELECTION CYCLE USING A PROBABILITY-BASED INTERNET PANEL
Trevor Tompson, The Associated Press
Mike Lawrence, Knowledge Networks; Email
Stefan Subias, Knowledge Networks; Email
The Associated Press and Yahoo! News are conducting a longitudinal study tracking the attitudes and perceptions of a national sample of more than 2,000 U.S. adults during the 2007-2008 election cycle. Between November 2007 and October 2008, the survey participants will be administered eight (8) online surveys, measuring the perceptions of these individuals from before the first primary through just before the general election in November, 2008. The survey participants are members of the probability-based internet panel created by Knowledge Networks. The study presents an opportunity to investigate sample and data quality for an online longitudinal sample. Traditional longitudinal studies use costly means for limiting attrition, such as substantial respondent incentives and interviewer visits and telephone calls to respondents. While the study utilizes relatively small panelist incentives, email reminders, and limited telephone followup, it emphasizes the opportunity for participants to share their evolving perceptions from well before the first primary or caucus up to the general election along with a limited opportunity to share "man on the street" perceptions with AP reporters. It will also include planned fresh cross-sections to test for panel conditioning. We will present analytic findings regarding methodological issues using survey response data and metadata from the first few waves of online data collection. The areas of inquiry are: (i) predictors of unit-level non-response (survival analysis across the waves); (ii) item non-response patterns and the impact of item non-response on survey results; (iii) panel conditioning effects on the survey response data. Substantive findings will also be presented bearing on the relationship between media consumption related to the election and survey results.
