Connecting Polls with Quality: An Interview with Sunshine Hillygus

As an Associate Professor of Government and Director of the Program on Survey Research at Harvard University, Sunshine Hillygus has a constant eye on best survey practices. Having published articles on voter decision-making and receptivity, she has an expert view on research design and methods that lead to quality results, especially as they apply to polling practices—a burning topic during our primary election season in the U.S. We recently spoke with Dr. Hillygus; our interview follows.

 

As the Director of Harvard's new survey program, what is your mandate?

Our primary mission is to provide survey research resources both for teaching and for research at Harvard; the second is to become an internationally recognized center for excellence in survey methods. We advise about survey research methods, encourage exchange among scholars, and we bring in practitioners to engage with scholars in areas like survey quality and questionnaire design. People often do not realize that there is an entire academic field of study for best practices and survey methods, and literature that provides guidance on survey quality. We also try to create opportunities for dialogue across academic disciplines.

What do you feel are some of the challenges to online survey research quality these days?

One of my motivations for creating the program in survey research was that I kept encountering misconceptions about online surveys. There has been a tendency for academics to assume that all online surveys are the same. I often find myself explaining that KnowledgePanel® is different and better than other online survey panels, because it is probability-based. One of the goals of our program is to create a set of quality standards that will make it easier for scholars, academic journal reviewers, editors, and journalists to gauge whether or not they should question the results because of the survey method.

Do you expect that academic and government research will transition from phone to online – is it "time"?

Clearly, there are problems with telephone surveys these days, particularly with quick turnaround polls that regularly get response rates under ten percent. But just because telephone surveys are flawed doesn't mean we should accept any flawed online survey. The survey research community, including practitioners and scholars, needs to come up with a set of standards and procedures for online surveys.

What should the role of the academic community be in leading the charge for more dependable survey results?

As I mentioned, the survey industry is facing some tough challenges, including lower response rates. New technologies are influencing whether people are willing to participate in surveys and are influencing general survey credibility. It's vitally important for academics, along with practitioners and government scholars, to work on solutions together. Academics need to be concerned about the impact of these challenges – not just on questions of survey research methodology, but on the substantive questions that span across nearly every academic discipline. It's not just going to impact a small group of people who are interested in survey methods; it has a wide impact on nearly any topic we study, certainly in the social sciences.

Do you feel that the electorate can distinguish non-scientific polls? Is it becoming important to them?

The credibility of the survey industry is affected by the general confusion about survey quality standards. It becomes easy for people to dismiss the entire industry out of hand due to the wide variation in survey quality. Mediating between the public and the pollsters are journalists, and they play perhaps the most critical role in shaping perceptions about the survey industry. The most important part of bridging the survey knowledge gap is to make sure that journalists are informed about survey quality standards. It's so tempting to jump on the bandwagon of any new controversial poll that comes out, without regard to its quality. A number of well-respected media outlets have instituted standards, and I applaud their efforts in making sure that they are only reporting high quality survey results. Organizations like ABC News and The New York Times are working hard to take survey quality into account, and I hope that more journalists and academics will follow their lead.

Thinking about the presidential primaries, would you offer any suggestions or advice to pollsters attempting to capture the issues and candidate preferences that seem to be shifting among the electorate?

Pollsters got a spanking coming out of the New Hampshire primary; that's unfortunate. With some guidance from political science, the media would have concluded that we shouldn't have expected to know the answer before the actual election. Both polls and voters are volatile in primary season, and the media should have been hesitant about declaring somebody the inevitable winner. Journalists have a responsibility to interpret polls in an appropriate way. Getting back to the issue of survey quality, it drives me crazy when a particular pollster or polling house is given the gold star of quality for correctly predicting election results; this isn't the right standard by which to judge survey quality. When we're dealing with close elections, as we've seen in recent years, frankly it's not hard to get close to the right answer since most results are in tight range of numbers – 45% and 55% in a close two-person race. And being closest to the right answer doesn't mean that the polling house made the best decision about their likely voter screen or their sampling procedures. There is research – in survey methods and political science – to back up particular polling decisions.

Do you feel that survey researchers can utilize an online methodology to understand and predict receptivity to these issues and to a candidate's campaign, and if so, are there any limitations?

Absolutely, my own research does so. The strength of polls and surveys is in explaining behavior rather than in predicting what will happen. It's unfortunate that there's so much focus on the horse race and less tendency to look at the mix of voter response to campaign information or to candidates, which is the real power of surveys and polls. Whether you're talking about the unique capabilities of an online poll to show video or pictures, or other aspects unique to online surveys, there are all kinds of things that we can learn about an election from online surveys.

Do you see Knowledge Networks as having unique capabilities and advantages in terms of online methodology?

The unique advantage of Knowledge Networks is how the recruiting methodology is different from every other online survey; I spend a lot of time trying to explain why online surveys are not all the same. The online mode does offer unique advantages to telephone surveys. In my area of study – political campaigns – the ability to show television ads on screen has contributed greatly to our knowledge about the influence of television advertising on voters. There are unique capabilities to online surveys for political science and unique methodological advantages to Knowledge Networks over other online panels.

Do you agree that there's increasing political polarization in the U.S. among the electorate, and if so, what effect will it have on the electorate with regard to the upcoming election?

I think there is an overblown notion of polarization. The Republican and Democratic camps are both diverse and complex groups of individuals. The polarization perception is a reflection of the fact that political choices are more polarized, and that political elites are increasingly polarized. Plus it makes a heck of a better story to talk about conflicts than to talk about the lack of conflict.

Do you feel that there are specific factors that contribute to the lack of viability for third party political candidates in U.S. politics?

It makes for a great headline to talk about Bloomberg considering a run, but the reality is that for a minor party candidate in the American political system, there are enormous barriers – both institutional and psychological. In many states, for instance, to even get on the ballot as a third party candidate, you need a petition within a certain amount of time signed by at least five percent of voters in the previous election. A candidate also needs specific levels of support to participate in debates and receive federal matching funds. We have to remember that Democrats and Republicans have decided on the rules of the game, so they've put up institutional barriers that make it more difficult for minor party candidates. The Electoral College system is winner take all, so it's difficult for a candidate to maintain any level of support. Perot won 19% of the popular vote but didn't get a single Electoral College vote, so even though he had this high level of support in 1992, it was difficult for him to sustain and build on that support in the following presidential election. There are also all kinds of things about the campaign itself that reinforce the two-party system, such as rules about debate participation, matching funds, and unequal news coverage.

The Electoral College seems to be something that few American citizens understand.

We have one of the most complicated electoral systems in the world. The Electoral College for the general election certainly creates barriers for minor party candidates, but that's to say nothing about the rules of the game for the nomination process. I did an interview with the Australian Broadcasting Corporation trying to explain the presidential primary system, and the journalist eventually concluded, "You just have a crazy system." I totally agree, which is why using polls as some predictor of outcomes is so difficult in a primary. In part, this is because poll results actually influence how people will behave, which makes it difficult for a snapshot taken even a few days before an election to be accurate. Again, while polls and surveys serve a very important function in all elections, I think their primary function is to explain rather than to predict.

With an ever-increasing number of media platforms available to political campaigns, do you foresee an upward trend in the number of voters in this presidential election?

I wouldn't necessarily connect the number of media platforms with voter turnout. The fact that this general election is an open seat race will make it very competitive, which I expect will contribute to higher levels of turnout. But the question is, "higher levels than what?" It's possible that we won't exceed the last election.

But media choice does influence election politics. If you were watching television in the 1960s while eating dinner, you had no choice but to watch the nightly news. Today, due to media expansion, it's easier for people to stay isolated from politics. The behaviors and efforts of the candidates also reinforce this. Most people don't realize it, but your registration data is public record, and the candidates and parties use that information – your name, address, gender, party, and turnout history – in their campaign strategies. Parties use this information to spend their resources more efficiently by targeting individuals with a strong voting history. Over recent election cycles, the gap in campaign effort going towards those who are active voters and those who are not registered has absolutely polarized. Unregistered voters are not receiving campaign appeals. We know from political science that one of the biggest predictors of whether someone votes is if they're asked to do so. Today, candidates only knock on the doors of individuals who are likely to vote. That reinforces turnout among the politically involved class of people in the electorate, and ignores the unengaged, meaning they are less likely to ever become engaged.