The Truth: Why Online Sample Sucks for Voter Opinion Surveys

As a political pollster I often get asked by clients and the media if Magellan Strategies uses online sample for voter opinion surveys. This is a very good question, and shows that people are aware of the rapid decline of voter households with a landline telephone, and the growth of “cell phone only” or “cell phone mostly” households. Many people think it should be relatively easy for a political polling firm to simply email a bunch of voters and ask them the same questions that we do over the phone.  Unfortunately that is not the case, and believe me when I say we have tried many different approaches to get registered and likely voters to participate in a survey by emailing them an invitation. The truth is that the available online sample sucks for a voter opinion survey. Let me explain the reasons why.

Tablet online survey.

For a voter opinion survey to be accurate, it must reflect the demographics of the voter population that the survey is attempting to measure. When we randomly draw survey sample from our voter files, the cell phone and landline numbers we are about to call are representative of the entire eligible voter population. Assuming the voter file is thoroughly phone matched with landline and cell phone numbers, the randomly drawn sample will contain the proper amount of phone numbers needed to interview Democrats, Republicans and independent voters. It should include plenty of phone numbers for Hispanic and non-Hispanic voters, as well as voters of all ages. This time-tested approach to using landline and cell phone sample for political polling can be applied to any level of geography, including national, statewide, Congressional, county-level and legislative district surveys. As long as I have at least around 3,000 unique phone numbers to call, I can get a decent read on voter opinion.

In contrast, the capabilities of using online sample for a political poll or voter opinion survey are limited.

After the 2012 election, we were determined at Magellan Strategies to figure out if online sample could be an acceptable  proxy for telephone sample among hard to interview voter populations such as younger and minority voters. We were also very curious if we could field an accurate survey of likely primary or general election voters using 100% online sample. So we set out contacting the biggest and best online panel companies in the United States. We ended up working with three different companies, Survey Sampling International (SSI), Usamp (rebranded to Instantly, Inc.) and Research Now.  We then sent them our Iowa, North Carolina, Pennsylvania and Colorado voter files and asked them to match the registered voters to their database of online panelists. This process would give us a good sense of which of the panelists were voters, and allow for us to know the demographics of those “voter panelists”.  Unfortunately what we found was discouraging.

For starters, there were not enough panelists in each state to support our preferred sample size of 500 interviews. When we tested the sample with a “standard” voter opinion survey asking questions like candidate image rating, job approval and ballot test questions, we could not come close to 300 interviews. The other disappointment was that the unweighted results showed respondents being overwhelmingly female, over-educated, unemployed and older than 44 years old.  Our hope was that we would get a more balanced distribution of respondents that were male and female, Hispanic, and under the age of 44. It also took a lot longer for enough respondents to complete the survey, even after several reminders from the panel company.  This was a huge drawback to the method. In September or October of an election cycle there is not enough time to wait five days for results.

Moving forward three years from our online panel tests, in 2016 we reached out to the same panel companies and asked them if they more panelists had joined, basically asking if their panels had “matured”. I asked what it would take to complete a 500n survey of likely 2016 voters in Colorado. The company said we would likely have to expand the universe into other states like New Mexico or Arizona. The fact is that it is too difficult to use only online sample to conduct a reliable voter opinion survey within a state or political district. It simply takes too long and the demographics of the voters who respond are just too far off from the demographics of the likely voting population.

However, I must say that online sample is representative enough for a national survey of registered or likely voters. We have fielded a couple of national online surveys and have found that the best approach is to combine online panels from two different companies. I will also say that there are other political pollsters who do use online interviews as a part of their overall sample mix (the other interviews being among landline households) for a state level survey. Rather than use an online panel to send the survey, they do an email blast to a really large number of younger voters. They have found that this method is acceptable enough for their purposes.

In summary, in a political pollster’s perfect world we would be able to email our surveys to registered and likely voters and have everyone participate. However, that is just not reality and we are a long way from this being a realistic possibility.

By | 2017-05-24T03:25:53+00:00 July 21st, 2016|