Political candidates, campaign professionals, political observers and the media refer to automated voice recorded surveys in many different ways. These include “IVR (interactive voice recorded) survey”, “flash poll”, “robo-poll”, or a “push-button poll”. These connotations do not exactly inspire trust and confidence in the survey method to measure voter opinion accurately. However, as a polling and survey research firm that has fielded thousands of automated voice recorded surveys to measure voter opinion since 2009, we have a strong understanding of the automated interview method’s strengths and weaknesses. The fact is that automated voice recorded, or IVR, surveys measure certain voter populations very well and others very poorly. Like all survey interview methods there are strengths and weaknesses. Let’s first cover the strengths of automated voice recorded surveys.
The Good: Large Sample Sizes
Automated surveys are able to generate more interviews for a lower cost than live interview landline or cell phone surveys. Depending on the amount of landline sample available for the state or district, we always aim to complete 700 interviews, which has a margin of error of +/- 3.70%. The large sample also allows clients the ability to drill further down into subgroups of the voting population (men/women/Republicans/Democrats/Independents) with a higher confidence interval. For example, a voter survey of 400 respondents would yield interviews of approximately 200 men and 200 women. A 200n sample size has a margin of error of +/- 6.93%. However, a survey of voter survey of 700 would yield interviews of approximately 350 men and 350 women for a margin of error of 5.24%. The larger sample size allows campaign managers and decision makers to make better decisions about targeted messaging to voter subgroups with more confidence.
The Good: Automated/IVR Surveys are More Affordable Than Live Interview Surveys
Polling and survey research firms that offer automated voice recorded and IVR surveys will vary in their pricing, but the bottom line is that automated voter opinion surveys cost significantly less than live interview surveys. Our standard pricing for an automated voice recorded voter opinion survey is $3,500 for 15 total questions and 700 interviews. That same sample size and number of questions for a live interview survey using landline and cell phone sample would be more than $10,000.
The Good: Automated/IVR Platforms Are Maturing
When we first started doing automated voice recorded/IVR surveys we used platforms that were very difficult to work with. We would record each prompt, then login to the platform and piece together the survey and set up the different prompts. The process to set up a single survey was cumbersome, time consuming and inefficient. Today the platform we use is much easier to work with. We are now able to record one question and use it again in other surveys, so we only have to record that question once. We are able to set quotas so that the survey automatically stops after a certain number of interviews, and we can randomize response options within a single question, and randomize a set of individual questions. These advances help reduce response bias and generate a higher quality voter 0pinion survey. The biggest advancement that we really love here at Magellan is the ability to capture verbatim responses, which is crucial to understanding why voters feel a particular way about a candidate or issue.
The Good: Automated/IVR Surveys Still Reliable for Republican Primary Elections
In case you have never done the research, the typical Republican Primary voting population anywhere in the country is extremely white and old. Voters who are 65 and older usually make up 35% or more of the likely Republican Primary voter universe. In some states, the 65 and older population exceeds 50% of the vote in a Republican primary. And again, the demographics of the population who still have a landline telephone are extremely white and old. This is why an automated voice recorded survey is reliable and acceptable for voter opinion research among a Republican Primary voting population.
The Bad: Poor Coverage of Younger and Minority Voters
What is the worst thing about an automated voice recorded survey? It is nearly impossible to obtain a good sample of voters 45 or younger, Latinos and African Americans. Even when the best landline telephone sample is used, there are just not enough voter households among voters 45 and younger to get a good read. Our unweighted automated voice recorded surveys typically will be more Republican, older and whiter than the voting population we are trying to measure. Understanding the demographics of individuals who have a landline telephone is important because you can get a clear picture of the voters who are not going to be in your sample. That is why automated surveys have significant problems in multi-ethnic populations like California, Nevada, Florida, Texas, New York and states with large Hispanic populations including Arizona, New Mexico and Colorado.
The Bad: Automated/IVR Surveys Struggle to Interview Moderate Voters
It is our belief that voters who participate in automated voice recorded surveys tend to be more intense on both sides of the political spectrum. We cannot prove this hypothesis, but after fielding 3,000 automated/IVR surveys we get the sense that moderate voters who are not interested in politics or policy issues are less likely to participate. However, the motivated, intense progressive, liberal, conservative or Tea Party voter wants their voice heard and so they are eager to participate in an automated voice recorded/IVR survey.
The Bad: Automated/IVR Alone is Not Reliable for a Presidential General Election Survey
Because the percentage of younger voters (18 to 34 years old) who participate in a Presidential election has increased from 12% to 22% in most states, the likely voter population “coverage” of an automated voice recorded survey is not sufficient. In other words, it simply will not interview enough younger voters to accurately measure voter opinion. Because of this problem we often field hybrid surveys of 30% cell phones interviews and 70% automated voice recorded/IVR surveys. Over time, the likely voting population “coverage” problems will only increase for automated voice recorded/IVR surveys and eventually they will be completely unreliable, unless laws are changed to allow for calling cell phones.