Polling & Research Quality Circa 2008: An Interview with Kathy Frankovic

As Director of Surveys for CBS News, Kathy Frankovic has a deep investment in the quality of consumer research; her work reaches millions of people via TV, radio, the Internet, and other media, and she is repeatedly called on to accurately measure the public's opinions about candidates and key political issues. She is a co-author of The Election of 1980 and The Election of 1992, and has published many articles on elections and public opinion. With the presidential race of 2008 already so much on the public's mind, we thought it was the perfect time to talk to Frankovic about the key issues of research quality as she sees them.

What is your sense right now of the awareness among the people you deal with of research accuracy?
I work for a news organization, and that means that we have to be accurate; when we are not, we are punished in public opinion. Anything we report has the CBS News name on it – and if our name isn't associated with accuracy in reporting, then we have a problem. I think probably even more so than a traditional corporation, we have a huge concern about accuracy and data quality.

What about the market research industry—is there a bigger awareness of quality issues compared to five years ago?
That's an interesting question. Certainly the concerns that all researchers have about response rates and all sorts of data quality issues have become much more visible. I think people have articulated them more. I did an article after the 2004 election, and one thing I pointed out is that even news reporters raise issues about public opinion polls in a way that they never used to. There were news stories about response rates, about cell phones – whether there are just too many cell phones out there to draw a random sampling of the public. So I think in many ways data quality concerns have increased everywhere.

How do you feel online research has matured in recent years?
It does seem to be much more acceptable in market research. What I do is to look at the voting public overall, and there are still difficulties with coverage for that application. The representation question still exists – that is a real issue. That is why CBS works with Knowledge Networks; we know that we are dealing with a randomly selected sample of the public, some of whom are even cellphone-only households. We've utilized this successfully several times – particularly for immediate reactions to events.

How do you feel that the public's attitude toward research has changed?
I think there is a longterm change. Twenty or thirty years ago, the assumption was that there were objective measures of what the public thought, and that politicians needed to be aware of, and even respond to, opinion polls. If you look at speeches of candidates in the past who were confronted with negative polls, they would take them seriously.

In the 1990s and the current decade, we are seeing polls treated in a different way. They are viewed as potentially partisan, so that politicians now don't necessarily take a negative poll to heart; they will first attack the research itself. So there is a greater combativeness when it comes to dealing with polling information, and there has been a greater assumption of partisanship.

There are a lot of reasons for this – the increase of polarization in politics, the availability of a lot more resources – online blogs, talk radio. I think all of that kind of slides over into how the public views research in general. The public may be viewing public opinion polls one way, but they can't help but link them with other kinds of polls and research.

So given that environment, that skepticism, how do you assure that CBS News polls are seen as reliable?
We have put a lot of emphasis on understanding our methodology. We have been good at disclosing the relevant information, so that people know how to judge a poll. We keep on doing them, keep on reporting them. I think that we are very conscious of the criticism we might get if someone does not like a result that we report.

What about your own decisions regarding research quality – what makes you feel that a given poll or methodology is reliable enough to be used on the air, to carry the CBS News name?
Well, you hit the nail on the head: "Are we willing to put our name on this?" I think reporting it is one thing, putting our name on it is something else. The work that we do with Knowledge Networks is a "CBS News Poll"; we are saying we are confident enough in what we have done that we are going to call it a CBS News product.

We do have polling standards, and they specifically talk about concerns about online polls. They say, among other things: "Surveys that are inherently not representative of a population should not be reported to be indicators of opinion." They also cover wording, when polls are conducted, and other issues.

What do you think the effect of the Internet has been on trust in polling?
I think people expect to have opinions immediately. I don't know whether the more polls you see, the better – or worse – you feel about polling. Clearly, you see more things that call themselves polls, even if they are not.

And do you think that people make a distinction between a quick poll and more serious research?
There is some data on this; a lot of people don't see any difference. There are also a lot of people who say, "The more respondents, the better" – it doesn't matter how they are selected. It's not at all clear that people understand what it is that an opinion poll is, and what's good and bad about them.

You have used the KN Public Pulse technology to get immediate opinions; was it the probability sample that drew you to that approach?
Absolutely. Having an RDD-based panel, which represents the entire country, is very important to us. We obviously restricted participation in our KN Public Pulse for the debates to people who said they were going to be watching a debate. We have been able to report results within half an hour at the end of the debate, which is just stunning. Clearly your panel has been extremely useful to us in allowing us to report the public ongoing reactions to debates and other political events.

Are there things you would like to be able to do in research that you can't do at the moment?
Sure. I would like to be able to have a really good survey in Iraq or Afghanistan; but you can't do that. I would love to be able to talk about the people who are really attentive to the political campaign for 2008 – but it's still a pretty small group of people. I really am focused on the election cycle.

Can you tell us any of your plans around the 2008 election?
We are still working that out. We are intensely interested in the debates, and in the primary period. When it comes to the Fall, we would like to do better estimates of the status of the Electoral College. That's all I can say now.

Are there certain segments of the population that you think are going to become more crucial for research in 2008?
I think it's still a little early to judge. I think it will be interesting to see what role military families play; but I keep telling people that it's still really, really early.

I am curious about whether you feel the election poll controversies are still spilling over into concern about polls generally.
I think there is no way they couldn't. Back in 1948, Harry Truman was going around talking about how the "sleeping polls" – like sleeping pills – were trying to lull voters to sleep and not to vote. On the day after the 1948 election, Roper and Gallup and Crossly had to spend a lot of time reassuring their corporate clients about the validity of what they were doing. So I think any criticism of public opinion polls spills over into the views of market research. They may not have to defend their polling work quite as much as we do, but they still have to be aware of it.

Do you have any other thoughts on research accuracy?
We always want to ensure that what we do under the CBS News name is accurate and reliable. We are going through a review right now of the CBS News Poll, which I think is great. This is the right time to do it. I think we would be foolish not to be alert to quality issues.