A Tale of Two Polls

By Monday, March 24, 2014 0 No tags Permalink 0

There are the best of polls. There are the worst of polls. Not all polls are created equal. Some are designed to gauge public opinion. Some are designed to form public opinion. How to tell the difference? One way is to check them against the 20 questions in The Public Agenda article 20 Questions Journalists Should Ask About Poll Results.

The first two questions produce interesting answers in the case of two polls measuring what the public thinks about having jets fly out of the island airport.

Question.1   Who did the polls?

In the case of the jets, Forum Research is one. The company is described as specializing “in political polling and customer satisfaction research. Some of the firm’s research tools include small and large-scale telephone, interactive telephone, and online polling, focus groups, one-on-one interviews, mail, door-to-door, intercept and personal surveys”. Forum Research methodologies are not without controversy.

Ipsos Reid is the other. Wikipedia says this about the company. “Ipsos Reid is Canada’s largest market research and public opinion polling firm. The company’s researchers conduct both syndicated and customized research studies across key sectors of the Canadian economy, including consumer packaged goods, financial services, automotive, retail, health and technology & telecommunications.”

A number of websites over the years have made observations about the company’s methods. That includes this blog.

Question 2   Who paid for the poll and why was it done?

Again from Wikipedia: Forum conducts regular polls on municipal, provincial and federal politics for major Canadian news outlets, including the Toronto Star,National Post and Sun newspaper chain.

Ipsos Reid produced polls for the Toronto Port Authority(TPA) in 2012, 2013, 2014. The 2013 and 2014 polls had questions about jets.

The TPA cited results of the 2013 poll (Instances of the Toronto Port Authority citing the Ipsos Reid 2013 poll) since September 2013 including in its letter to Jim Flaherty asking for $100,000,000 and in TPA president Geoff Wilson`s presentation to the Globe and Mail editorial board on February 21, 2014.

The results of the polls raises questions of their own.

The first set of results, Table 1, come from Forum Research.

Date of Poll Support Jets Don’t Support Jets Don`t Know (DK)
April 2013 47% 37% 16%
July 2013 51% 35% 14%
September 2013 49% 36% 15%
December 2013 43% 39% 18%
March 2014 46% 40% 14%

Table 1

 

The results shown in Table 2 below are from both Ipsos Reid`s 2013 and 2014 polls.

Date of Poll Support Jets Don’t Support Jets Don`t Know (DK)
July 2013 60% 37% 3%
January 2014 61% 35% 4%

Table 2

 

Table 3 compares the results of each company’s polls at the months closest to their execution.

Date of Poll Support Jets Don’t Support Jets Date of Poll Support Jets Don’t Support Jets
Ipsos Reid July 2013 60% 37% Forum July 2013 51% 35%
Ipsos Reid January 2014 61% 35% Forum December 2013 43% 39%

Table 3

 

How is it that the Ipsos Reid results for jets are 9% higher in July and 18% higher in January than the Forum Research the results for comparable time periods despite the +/- 3 %, 19 times out of 20 margin of error stated for each?

 

One clue may be in the way the question was asked. According to 20 Questions Journalists Should Ask About Poll Results, the way the question is worded can affect the results.

  • This is the Forum question13. “Would you approve or disapprove of allowing jet aircraft to use the Billy Bishop Toronto City Centre Airport?”
  • This is the Ipsos Reid question. “Some people support the use of jet aircraft at the Billy Bishop Toronto City Airport and say that it would increase competition and lower prices for flights out of Toronto, give Torontonians more access to more destinations, and help the Canadian economy and aerospace companies like Bombardier due to increased manufacturing orders for jet aircraft. Other people oppose the use of jet aircraft at BBTCA and say that allowing jets to fly out of Billy Bishop Toronto City Airport would lead to more noise coming from the airport and skies, require a runway extension, increase traffic congestion due to higher passenger volumes, and open the door for other airlines to fly jets out of the Billy Bishop Toronto City Airport. Which is closer to your point of view?”

 

Another clue may be in the way the margin of error is reported.

A comparison of the data from one region, North York, Table 4 is a case in point.

Ipsos Reid – North York Results Forum Research – North York Results
Date of Poll Support Jets Don’t Support Jets D K Date of Poll Support Jets Don’t Support Jets D K
July 2013 58% 40% 3% July 2013 55% 31% 14%
January 2014 61% 36% 3% December 2013 51% 26% 23%

Table 4

Forum states that the margin of error for the total sample is considered accurate +/- 3 %, 19 times out of 20. Subsample results, Forum says, will be less accurate. Although they are not reported, subsample (such as region) are available at www.forumresearch.com/samplestim.asp for anyone wishing to pursue the matter.

Ipsos Reid, on the other hand, states that the margin of error for regions such as North York is ± 11.3 percentage points. In other words, the good people of North York could have supported jets in January 2014 by as much as 72.3% or as little as 47.3%, a number more in line with Forum`s December number for its total results. However, a 47.3% approval rating might not have fit in with the TPA’s need.

 

As the article 20 Questions Journalists Should Ask About Poll Results points out there are other factors at play. Weighting and question order also affect poll results, especially when an organization uses them to achieve a goal.

As Ipsos Reid advertises on its website, “We understand and manage issues”(emphasis added).

 

 

 

No Comments Yet.

Leave a Reply