From July 24 to August 5, 2012, Ipsos-Reid, third ranked research company worldwide, conducted a poll for their clients, the Toronto Port Authority.  The port authority released the results in August to great fanfare.  Not surprisingly, the joint port authority – Ipsos-Reid press release read, “Survey Reveals Most Torontonians Agree Billy Bishop Toronto City Airport Great for Toronto’s Economy”.

Should we be surprised?

First, something about surveys.  Several websites discuss the construction of surveys and point out some of the pitfalls in their design that may skew results, intentionally or not.  Of course, design faults may be used intentionally to produced a desired result.

Most of the websites and their tutorials list three main design faults: question order, leading questions; and loaded words.

Question Order

Question order can effect responses and so skew results.  The respondent’s answer to the next question may be affected by the preceding question. One way around this is to order the questions by topic, all in one section.

The Ipsos-Reid survey deals with the Billy Bishop Toronto Centre Airport in questions Q11, Q12 and Q13.  Then, Q15 returns to the airport after Q14.  Q14 deals with the port authority.  There is a reason for this.

Leading Questions

Q14 asks respondents if their impression of the port authority is improved or worsened after each of the following statements.  What are the odds that a respondent are going to have a worsened impression after the statements?

Removes pollution and contaminated garbage from the Don River and the inner harbour

Created a wetland in cooperation with conservation authorities to protect wildlife habitats along the waterfront

Is making long-term investments in improving environmental sustainability practices.

Operates Billy Bishop Toronto City Airport, used often in medical emergencies by Ontario’s air ambulance service.

Purchases all of its power from renewable sources.

Ensures safe and efficient movement of ships in the harbour, including sailboats.

Sponsors festivals and exhibits within the harbourfront community

Is involved in creating a fish habitat at Tommy Thompson Park.

Works with the city and all levels of government to build the tunnel that connects Toronto Island with the mainland

Supports at-risk youth through the Room 13 leadership program at the Harbourfront Community Centre.

Regulates and oversees recreational boating in the Harbour of Toronto

Provides transportation, distribution, storage and container services to businesses Is behind Sail-In Cinema, where movies are projected onto a barge in the harbor near Sugar Beach and can be seen from land and nearby boats

Owns and operates Billy Bishop Toronto City Airport.

Carries out dredging prevent flooding around the Don River.

Now that the respondents have been softened up with all these motherhood issues, the pollsters drop Q15 on them.  Again, it’s a multiple-part question.  Interestingly, it starts off with the port authority, that feel-good organization that does so many good works.  Perhaps the fear is that the respondents need their good will pumped before answering these.

Q15.  Please tell me whether you strongly agree, somewhat agree, somewhat disagree, or strongly disagree with the following statements

Q15_1. [The Toronto Port Authority provides important economic benefits to the City of Toronto]

Q15_2. [The Toronto Port Authority is a forward-thinking organization]

Q15_3. [Billy Bishop Toronto City Airport is an important part of Toronto’s future economic growth.]

Q15_4. [Billy Bishop Airport is an economic engine of the city]

Q15_5. [The island airport makes a great first impression for visitors who use it to travel to Toronto]

Q15_6. [The Billy Bishop Airport is great for the economy of Toronto]

Q15_7. [The Billy Bishop Airport is a great gateway for the city and tourists]

Q15_8. [The Billy Bishop Airport is a valuable asset for the city]

Note how many times the word great is used in these questions?  It’s a great word to take us on to the next section: loaded words.

Loaded Words

Here’s what one tutorial said about loaded words.  “Loaded words are those which carry overtones or connotations that predispose a survey respondent to think in a certain way.  If researchers use overly strong words (positive or negative), or use labels with clearly judgmental implications, this can bias respondents by priming them for certain mindsets.”

See if you can spot any loaded words in this question, with three possible responses, about the airport.

Q13. And thinking about the Billy Bishop Toronto City Airport, which of the following would best describe your attitude toward the airport and its operations?

I’m personally dead set against having an airport on the island and believe it’s bad for Toronto

I’m personally in favour of having an airport on the island and believe it’s good for Toronto

I really don’t care one way or the other about having an airport on the island

It makes you wonder if different wording, say like that of Q4 would have given a different result.

Q4. How important is improving Toronto’s waterfront to you? Is it…

Very important

Somewhat important Very important

Somewhat important

Not very important

Not at all important

(DK/NS)

Substitute the island airport for ‘improving Toronto’s waterfront’ and look for loaded words.

Q4. How important is the island airport to you?,

Very important

Somewhat important Very important

Somewhat important

Not very important

Not at all important

(DK/NS)

Perhaps if the survey had been designed a little differently, the port authority’s press release would not have read, “Survey Reveals Most Torontonians Agree Billy Bishop Toronto City Airport Great for Toronto’s Economy” but then what would have been the point of hiring Ipsos-Reid?

As for Ipsos-Reid, here’s an interesting claim on their website about what they do.

“We understand and manage issues, advance reputations, determine and pinpoint shifts in attitude and opinion, enhance communications, and evaluate policy.”

How does their claim impact on their port authority survey?  The go-to-it guys John Wright  email: or Sean Simpson, email: might be able to say.