Every January the National Mall in Washington, DC, fills with Catholic school students in matching neon hats holding signs and chanting, “We” -young people -“Are the Pro-Life Generation.”

This pervasive anti-abortion slogan popular at the annual March for Life refers to a 2016 poll commissioned by Students for Life, an anti-abortion group, which found that 53% of millennials (18- to 31-year-olds, the poll says, which most of the students holding those signs are not) think abortion should be illegal in all or most circumstances. But in that same poll, in the question right below it, a greater percentage of the respondents identified themselves as “pro-choice” (48%) than “pro-life” (36%).

The contradictory findings highlight the problems of abortion polling more generally: Abortion is such a complex and sensitive issue that few polls accurately capture how Americans really feel about abortion, but many easily manipulate the answers to support their cause.

BuzzFeed News analyzed 14 polls on the public’s opinion on abortion from the past four years conducted by a range of nonpartisan, advocacy, and media organizations, and spoke with researchers and representatives from eight of those groups. All of the researchers and most of the polls agreed: The public’s opinion on abortion is much less binary and significantly more complex than the national conversation about abortion makes it seem, and that accurately polling on abortion is particularly tricky.

“I don’t think any polls out there accurately portray the way the US population thinks about abortion,” Janine Beekman, an associate research scientist for the consulting and research firm Ipsos, told BuzzFeed News. “Abortion, more so than almost any other issue, is incredibly difficult to poll on.”

There are many reasons for this, researchers told BuzzFeed News, but three in particular: the intense emotional and political charge of the subject, the extreme knowledge gap surrounding the issue, and the lack of clarity on how opinions on abortion affect voting behavior.

Beekman and Ipsos recently conducted a study examining how difficult it is to gather people’s real feelings on abortion. They found, for example, that changing only one word in a question -“should abortion be legal” versus “should abortion remain legal” -could often result in opposite answers. They also found that asking people questions about what they think of specific abortion laws -common questions in some of the most cited polling by nonpartisan research organizations -often had little bearing on how those people actually felt about abortion itself, or on how they vote.

The Students for Life poll is a prime example. “We all know that a poll is only as good as the way questions are worded,” Matt Lamb, the group’s communications director told BuzzFeed News. “This poll shows when there are clearly defined questions, such as support for specific legislation or specific limits that people take the pro-life position.”

Polling giants Gallup and Pew annually ask the question, "Do you think abortion should be legal in all cases, legal in most cases, illegal in most cases, or illegal in all cases?" And every year they receive similar numbers in response. But Ipsos found that once you dig into those answers, the views get much more complex and contradictory.

For example, in one of the questionnaires, 76% of respondents identifying as Republicans agreed or strongly agreed with this statement: “We need to protect the rights of the unborn.” But later in the poll, 49% of Republicans agreed or strongly agreed with this statement: “I do not believe the government should prevent a woman from making her own decision about whether or not to have an abortion.”

“I think people who are answering the questions don’t quite know how they feel about abortion,” Beekman told BuzzFeed News. Though many poll-takers will initially seem confident in their answers, Ipsos found “it was clear that people, when discussing [abortion] in focus groups, really changed their attitude when they were forced to think deeply about it and see the nuance,” Beekman explained.

Ipsos isn’t the first group to study this. In 2015, PerryUndem, a research and communications firm, released a study that had very similar findings. In her article on the study for Vox, the organization’s cofounder Tresa Undem wrote, “having surveyed thousands of people across the country on the topic … [t]he biggest lesson I’ve learned is this: the current polling fails at accurately measuring opinion on this complex issue.”

Several of the questions in the PerryUndem/Vox poll have a pro–abortion rights slant, but the poll also asked Pew and Gallup’s questions on legality and got similar numbers, but when they asked the people who answered that abortion should be “illegal in most cases” if they think Roe v. Wade should be overturned, 53% of them said no.

“We don’t have a good idea of the private feelings on abortion. We have a good idea of the public thought,” Beekman said, referring to thoughts people feel comfortable expressing to a pollster. “The problem is, people vote with their private feelings, not their public thought.”

“Abortion is supercharged”

Just the word alone is shocking to some people. Up until just a few years ago, it was rare for the topic of abortion to appear in TV or in movies, especially as a nontragic plotline. Ever since the battle over legalization hit the mainstream in the 1960s and ’70s, politicians, advocates, and religious leaders have politicized nearly all of the language surrounding the issue, creating terms like “partial-birth abortion” or “reproductive freedom” and imbuing them with moral power.

This heightened sensitivity means that a lot of factors can affect polling answers, more so than many other issues. For example, researchers told BuzzFeed News, if respondents are answering the questions by phone when someone else is in the house they could provide different answers than they would online or alone. They may give different answers to a pollster who sounds like a young woman than one who sounds like an older man. If the group conducting the poll is an interest group like Planned Parenthood or the anti-abortion group Susan B. Anthony List, for example, and says that on the call, the poll-taker is more likely to give them answers favorable to the group’s point of view. Their answers are also often affected by -and reactive to -the pervasive media narrative at the time.

“During the Obama administration,” when the administration’s agenda was to make abortion more accessible, Barbara Carvalho, the director of the Marist Poll, said that polls started showing a slim majority “identifying as pro-life. … But now that there has been so much public debate that laws are going to change with restrictions, we see that the majority of Americans identify as pro-choice.”

Each organization that spoke with BuzzFeed News had a different way of trying to manage these variants through question phrasing and methodology: Marist, which conducts annual polls on abortion sponsored by the anti-abortion Catholic group Knights of Columbus, doesn’t reveal their sponsorship until the end of the call. NARAL, a pro–abortion rights group, sometimes uses a method of polling where they ask something similar to “What do you think your neighbor thinks about abortion?” NARAL’s Vice President of Communications and Strategic Research Adrienne Kimmell told BuzzFeed News. This is a strategy often used with topics like racism or sexuality that are considered “socially undesirable,” the idea being that what people say their neighbors think is what they themselves actually think, but do not want to admit. But other researchers argued that kind of question introduces a lot of “unknowns” into the polling and is potentially unreliable.

The sensitivity of the topic also makes it easy for interest groups to manipulate the questions to get the kinds of answers that suit their cause. A good example is polling on the Hyde Amendment, a federal rule that prevents any government funding from going toward abortion (with exceptions for rape, incest, and the life of the mother). This poll partnered with the abortion rights advocacy group All Above All asked, “As long as abortion is legal, the amount of money a woman has or doesn't have should not prevent her from being able to have an abortion,” and received 72% agreement, while Marist’s poll in partnership with the anti-abortion group Knights of Columbus asked, do “you strongly support, support, oppose, or strongly oppose using tax dollars to pay for a woman’s abortion,” and found 60% opposition or strong opposition.

“You can’t summarize abortion with any single question because it is sensitive to question wording. If you’re pushing the health button you get one thing; if you push the baby button you get another thing,“ Lydia Saad, senior editor at Gallup told BuzzFeed News. “It is pretty easy for [interest groups] to manipulate polls because it is very predictable, if you ask a question a certain way you can often know the answer.”

The knowledge gap around the issue of abortion also makes the answers murky. In the PerryUndem poll, 30% of respondents weren't sure what Roe v. Wade was about or thought it pertained to another topic besides abortion. A poll conducted in March and April 2018 by the Republican pollster Probolsky Research with the abortion rights group Guttmacher Institute showed that of the 1,000 people across the country who were asked “Based on what you know, is it legal for a woman to get an abortion in your state?” more than one in five people either thought abortion was not legal (8%) or were not sure (15%). (The poll has been written about by one of the analysts, and its findings and methodology were confirmed to BuzzFeed News by head of Probolsky Research, Adam Probolsky, but the data itself has not yet been publicly released.)

Saad of Gallup, however, argued that even if Americans don’t “have all the facts” on an issue, “it’s still useful to measure people’s opinions and views no matter how much they know.”


"A persuasion tool"

Ipsos and Gallup separately found that abortion is low on voters’ lists of issues of concern. The polls asking people to identify as either “pro-life or pro-choice” are pretty much split down party lines. And while they can be helpful in aiding interest groups and candidates in honing their messaging, they are rarely useful in predicting how people will vote.

So, what is this abortion polling actually for?

“It’s a persuasion tool, unfortunately,” Beekman said.

Interest groups and candidates on both sides of the abortion issue use polling data to persuade people that their position is what the rest of America believes, and that those who disagree with them are going against the views of the people. Or they use survey results to fire up voters who already agree with the view the poll is showing, to show them that America is on their side and so their side is righteous.

“Every person has their own logic. We have to respect public opinion for what it is and really try to understand it,” Saad told BuzzFeed News. “It’s sad to see polling used as a political football all the time, but there you have it — that’s political theater. And political theater is not where public opinion is.”