Understanding Survey Results: More than just Numbers
PUBLIC OPINION surveys are a way to assess what people think. It involves a methodology that has been proven accurate in tracking changes in attitudes and opinions.
Polls give the public a snapshot of a target population’s views on certain matters at a given point in time. Public opinion changes. Surveys help government and people keep the proverbial thumb on the public pulse.
Recently, the country’s top polling firms Social Weather Stations (SWS) and Pulse Asia released on October 8 and 13 respectively, the results of polls conducted in the last week of September on the people’s approval and trust of President Duterte.
SWS conducted its survey on September 23 to 27 with 1,500 respondents and reported a double-digit decline in the president’s public satisfaction and trust ratings, with a considerable drop in the satisfaction rating among the poor or economic classes D and E.
Presidential Spokesperson Ernesto Abella said the results were expected as the proverbial honeymoon period given to a new president had ended and argued that the drug war can’t be the sole reason for the decline. But Salvador Panelo, chief presidential legal counsel said the death of minors could have affected the ratings.
Pulse Asia’s survey, conducted on September 24 -30, revealed contrasting results. Media reported that based on 1,200 respondents, the poll showed that President Duterte still enjoyed a high level of approval and trust, with a rating of 80%—only 2% and 1% lower, respectively, than his June 2017 rating. Compared to SWS’s recorded declines, Pulse Asia showed only a minimal change in the way Filipinos of all socio-economic classes viewed the president and his performance.
Malacañang officials welcomed the release of the positive ratings, with a clear change in the tone of their statements. When asked about the Pulse Asia poll, Abella not only lauded the results but noted the failure of demolition jobs against President Duterte—a point which the spokesperson was not as vocal about when he commented on the SWS survey.
CMFR reviewed news reports in the newspapers (Manila Bulletin, the Philippine Daily Inquirer, The Philippine Star), primetime newscasts (ABS-CBN 2’s TV Patrol, CNN Philippines’ News Night, GMA-7’s 24 Oras, TV5’s Aksyon) and select news websites from October 8 to 14, 2017 and found little that could help the public understand why SWS and Pulse Asia had different findings.
Media’s reading of the ratings
Media treat poll numbers as hard news, something that should be reported as quickly as possible, highlighting the rise or decline in the numbers. CMFR observed this in the past, particularly during election season. In a Philippine Journalism Review article, journalist Romel R. Bagares said: “This ‘hard news’ mentality treats poll results as ‘facts’ that must be reported to the public the soonest time possible, and preferably, ahead of the others, instead of attempting to understand their implications to the democratic polity.” (“The Bad News about Public Opinion Polls,” August-September 2003)
In the case of the SWS survey, reports zeroed in on the huge decline in ratings, with experts saying that the shift could be due to prevailing issues during the survey period, such as the killing of minors in the government’s drug war and affecting mostly the poor.
But the Pulse Asia survey was not afforded the same treatment. Most accounts merely reported the findings; noting the contrasting findings of the SWS without discussing why the numbers were different and what the numbers meant.
Media made clear that SWS figures referred to net ratings; but failed to clarify that Pulse Asia numbers referred to gross ratings. In an email, Pulse Asia Research President Ronald D. Holmes told CMFR that the polling firm does not compute net ratings when reporting on performance and trust ratings, as it did in the case of the September 2017 nationwide survey which showed high approval ratings for Duterte. Holmes added that the net ratings could be computed by “simply subtracting the gross disapproval from the gross approval.” Such background should have been included in media accounts.
Poll data can help the public and officials form an opinion or rethink their position on pertinent issues. But for the data to be useful, media must do more than just go by the numbers.
CMFR cheers Rappler for an analytical report written by Carmela Fonbuena on the disparity between the results of the SWS and Pulse Asia surveys.
In “Understanding Duterte’s decline in SWS, Pulse surveys,” Fonbuena noted that the Pulse Asia survey numbers referred to gross satisfaction ratings, as opposed to the SWS’ net satisfaction ratings. Given this, the report warned against directly comparing the numbers from the polling firms’ respective surveys.
Rappler explained: “The net satisfaction rating is the difference between the satisfaction rating and the dissatisfaction rating, while ignoring the undecided. The gross rating ignores the percentage of the population who are dissatisfied and undecided.”
It observed that Pulse Asia has consistently recorded better numbers for President Duterte, noting that the former’s satisfaction rating (80%) is 13 points higher than that of its rival firm SWS (67%).
The report made an effort to explain the importance of the declines noted by SWS and Pulse Asia in the lower socio-economic classes. For example, Duterte’s 8-point decline in socio-economic class D in the September 2017 SWS survey to 68% from the 76% mark in the same period last year, while Pulse Asia showed a 7-point decline to 79% from 86%. The two surveys actually showed very similar findings on how the poor have changed their opinion about Duterte.
Pulse Asia chief research fellow Dr. Ana Tabunda told Rappler that Duterte “should look at his decline among the poor,” noting that the sector “make up the bulk of the population and typically get to elect presidents during elections” despite not being as active online as the more affluent economic classes.