Skip to content Skip to main navigation

News

Skilled legal advice with
accessible & personal attention

Behind the light rail survey

By Charlotte Harper - 21 October 2015 34

Survey methodology

Surveys into whether we Canberrans are for or against the ACT Government’s planned light rail system have been in the news all week, and many of our readers have questioned the research methods used.

We contacted Piazza Research, the Phillip-based company contracted to undertake surveys on light rail for the ACT Government, to find out how they operate and seek answers to questions like these from one of our readers:

“If a lot of the calls are made to mobile numbers (which are unlisted), so how does the pollster know which mobile number to call?”

“There must be a list provided of numbers that relate to Canberra people only so who is providing that list?”

“If random calls are being made to mobile numbers does this mean people in Perth or Darwin or Woop Woop are having their answers logged?”

Piazza Research director Grant Piazza says any research undertaken by his company has to be transparent and able to be critiqued.

He says Piazza Research includes mobile numbers in its sampling for phone surveys because there is a known age skew in using only landlines.

The pollster says his company uses an industry-certified supplier of phone numbers for research purposes, the same supplier used by market research giants like Nielsen and Newspoll. The name of the supplier is commercial in confidence.

The supplied numbers come initially from phone companies and are regularly tested using digital methods for geolocality and validity. Piazza says many mobile phone numbers arrive from the industry-certified supplier with geolocality data attached, but in any case Piazza Research asks for suburb of residence during the survey and removes those not in ACT from the sample.

To start with, Piazza Research receives a list of numbers that is much larger than needed then digitally selects a random sample to call. If skews are detected, they weight the data using a statistical procedure to remove the skew so results can be generalised to the ACT population.

The company stores caller data and responses in a secure server in Canberra. Those interviewed remain anonymous and are not identified during the research. Piazza Research does not identify individuals in any research project without their informed and specific consent.

The company is a quality certified social and market research organisation (ISO 20252) governed by rules set by the industry’s peak body, the Association of Market and Social Research Organisations (AMSRO). Piazza says ‘Mickey Mouse’ researchers can’t get into AMSRO. Piazza Research is also governed by its professional body, the Australian Market and Social Research Society (AMSRS). Every ISO-certified firm’s senior staff must be members of this organisation.

Piazza Research is audited every year to ensure their research methodology is correct.

As researchers, they don’t care whether the results are negative or positive for the client, they care whether the result is accurate and representative, Grant Piazza says.

Individual researchers may have views of their own, but when they design a research question it goes through 15 technical checks (to ensure it is useful, easy to understand, comprehensive enough and unbiased among other things).

Survey consultants design and check questions then a colleague checks and edits them. Often Piazza Research will run a pilot or test survey to verify that in a real situation people understand the questions and that the questions are appropriate.

Interviewers are trained. They’re not telemarketers and Piazza Research never engages in sales activity on behalf of its clients. It only collects information for research, Grant Piazza says.

 

 

 

 

 

 

What’s Your opinion?


Post a comment
Please login to post your comments, or connect with
34 Responses to
Behind the light rail survey
neanderthalsis 4:12 pm 22 Oct 15

Whenever a research company starts talking about how balanced their sample group is, I think of this:

A perfectly balanced sample:

https://www.youtube.com/watch?v=G0ZZJXw4MTA

VYBerlinaV8_is_back 12:57 pm 22 Oct 15

Ghettosmurf87 said :

wildturkeycanoe said :

As it is, the party in power didn’t even have the majority of the support of voters which in itself leaves most of Canberra not getting what they voted for.

I am so sick of this bogus argument being trotted out. The Liberal Party got 41votes more than the Labor Party. The Labor Party formed a coalition government with the Greens, who got 23,773 votes. Therefore, the current ACT government received 23,732 more votes than the Liberals.

The Liberal Party got 38.9% of the vote, the Labor Party got 38.9% the Greens got 10.7% of the vote.

On those numbers, no matter which party formed government with the greens, the situation would have been the same.

So in other words, the party that got 10.7% of the vote holds all the power.

Ghettosmurf87 12:36 pm 22 Oct 15

wildturkeycanoe said :

As it is, the party in power didn’t even have the majority of the support of voters which in itself leaves most of Canberra not getting what they voted for.

I am so sick of this bogus argument being trotted out. The Liberal Party got 41votes more than the Labor Party. The Labor Party formed a coalition government with the Greens, who got 23,773 votes. Therefore, the current ACT government received 23,732 more votes than the Liberals.

The Liberal Party got 38.9% of the vote, the Labor Party got 38.9% the Greens got 10.7% of the vote.

On those numbers, no matter which party formed government with the greens, the situation would have been the same.

dungfungus 12:09 pm 22 Oct 15

rubaiyat said :

I understand dungfungus that you are very concerned that governments only wastes money on things that you want them to waste it on 😀 but they do a lot of surveying and testing to ensure particularly their health campaigns are on target. Their anti-tobacco campaigns have been particularly effective and on target, in fact world beating. So much so that they have become the target of the Tobacco Industry and the same misinformation PR campaigns that you are susceptible to from the fossil fuel lobby.

Whenever I hear the term “world beating” my BS detector goes off.
I started smoking when I was 14 and gave it up when at maturity when I turned 21. I didn’t need an expensive tax-payer funded campaign to tell me smoking was unhealthy.
What happened to common sense?

rubaiyat 11:50 am 22 Oct 15

wildturkeycanoe said :

dungfungus said :

There is so much filtering of the target audience that there is no possibility of random polling as the pure meaning of that word means “in an unmethodical way” which is the opposite of how the polling companies operate.
I know you are passionate about a number of issues and I respect that but surely you can see the folly of these taxpayer-funded surveys.

Is it surprising to see the government spend our cash on a survey that is biased to support a project that the government is undertaking, in order to persuade us that we are the ones who want it?
I wouldn’t even be surprised to see comments from them such as “overwhelming support for light rail” or “dramatically changing trend towards acceptance of the tram”. Our elected representatives are simply telling us what we want rather than asking, because they would look very bad if true opinions on their performance actually came out in the mainstream media. For such a massive infrastructure spend a referendum should have taken place, instead of a massive campaign to try and get us to accept their decision. The election win was not a vote for the tram, it was more a vote against policies of the other parties and the tram was just a bitter pill Canberrans had to swallow in order to get the leadership they desired. As it is, the party in power didn’t even have the majority of the support of voters which in itself leaves most of Canberra not getting what they voted for.

Why are you so obsessed with having a referendum on the Light Rail but not on the greater sums spent on freeways?

Let me guess…

There may be a very good reason why the public is not seen as good decision makers on most projects.

You somehow haven’t managed to even understand that the government actually was voted in with a majority and a mixed agenda as it always is. What hope is there for an engineering planning proposal that would require at least some reading and concentration to make an informed decision?

wildturkeycanoe 11:03 am 22 Oct 15

dungfungus said :

There is so much filtering of the target audience that there is no possibility of random polling as the pure meaning of that word means “in an unmethodical way” which is the opposite of how the polling companies operate.
I know you are passionate about a number of issues and I respect that but surely you can see the folly of these taxpayer-funded surveys.

Is it surprising to see the government spend our cash on a survey that is biased to support a project that the government is undertaking, in order to persuade us that we are the ones who want it?
I wouldn’t even be surprised to see comments from them such as “overwhelming support for light rail” or “dramatically changing trend towards acceptance of the tram”. Our elected representatives are simply telling us what we want rather than asking, because they would look very bad if true opinions on their performance actually came out in the mainstream media. For such a massive infrastructure spend a referendum should have taken place, instead of a massive campaign to try and get us to accept their decision. The election win was not a vote for the tram, it was more a vote against policies of the other parties and the tram was just a bitter pill Canberrans had to swallow in order to get the leadership they desired. As it is, the party in power didn’t even have the majority of the support of voters which in itself leaves most of Canberra not getting what they voted for.

rubaiyat 9:18 am 22 Oct 15

I understand dungfungus that you are very concerned that governments only wastes money on things that you want them to waste it on 😀 but they do a lot of surveying and testing to ensure particularly their health campaigns are on target. Their anti-tobacco campaigns have been particularly effective and on target, in fact world beating. So much so that they have become the target of the Tobacco Industry and the same misinformation PR campaigns that you are susceptible to from the fossil fuel lobby.

rubaiyat 9:06 am 22 Oct 15

dungfungus said :

rubaiyat said :

As a clarification for people getting all concerned how surveys are conducted.

The professional companies’ reputation is on the line and they go to extreme lengths to get an accurate result within set parameters.

It is impractical to survey everyone and they full well know that people may not be truthful with their answers so they take appropriate measures to improve the statistics. First they over survey and ask cross checking questions. It is hard for people supplying false dat to keep their stories straight and they can be eliminated.

They also have excellent statistics as to what consists of a cross section of the target audience which is why they will ask you personal questions to set you up in the correct cohort. They already have information on you from the Bureau of Stats and the census as well as databases they buy from various data accumulators and retailers.

Lastly this is not about you personally. You may see yourself as the centre of the universe but nobody else does. You are just a sample and a very unlikely sample because unless the budget allows, a relatively small sample is all that is possible.

Obviously the larger the sample the more accurate the result but the survey companies are very good at getting useful results from smaller surveys. They qualify the results according to their calculated accuracy.

There is so much filtering of the target audience that there is no possibility of random polling as the pure meaning of that word means “in an unmethodical way” which is the opposite of how the polling companies operate.
I know you are passionate about a number of issues and I respect that but surely you can see the folly of these taxpayer-funded surveys.

The point is to get as accurate a result as possible.

That is why there is no randomness to the survey other than in the final actual selection of the candidates for interview. They call randomly within the selection without being able to control whether they get a response or a usable one. If they don’t they have to call again.

I have done these surveys when I was in university. The old adage of garbage in garbage out applies. The skill is in weaning out the garbage, which is what you are objecting to but if you don’t the survey is near useless.

Whilst it is also true that the client may want to phrase the questions to get the answers they want most times it is a very useful tool. Advertising companies use them all the time to the annoyance of the creatives. Which explains the mediocrity of most advertising these days. But advertising is there to manipulate and if it fails to do that it simply fails.

dungfungus 8:01 am 22 Oct 15

rubaiyat said :

As a clarification for people getting all concerned how surveys are conducted.

The professional companies’ reputation is on the line and they go to extreme lengths to get an accurate result within set parameters.

It is impractical to survey everyone and they full well know that people may not be truthful with their answers so they take appropriate measures to improve the statistics. First they over survey and ask cross checking questions. It is hard for people supplying false dat to keep their stories straight and they can be eliminated.

They also have excellent statistics as to what consists of a cross section of the target audience which is why they will ask you personal questions to set you up in the correct cohort. They already have information on you from the Bureau of Stats and the census as well as databases they buy from various data accumulators and retailers.

Lastly this is not about you personally. You may see yourself as the centre of the universe but nobody else does. You are just a sample and a very unlikely sample because unless the budget allows, a relatively small sample is all that is possible.

Obviously the larger the sample the more accurate the result but the survey companies are very good at getting useful results from smaller surveys. They qualify the results according to their calculated accuracy.

There is so much filtering of the target audience that there is no possibility of random polling as the pure meaning of that word means “in an unmethodical way” which is the opposite of how the polling companies operate.
I know you are passionate about a number of issues and I respect that but surely you can see the folly of these taxpayer-funded surveys.

rubaiyat 7:39 pm 21 Oct 15

As a clarification for people getting all concerned how surveys are conducted.

The professional companies’ reputation is on the line and they go to extreme lengths to get an accurate result within set parameters.

It is impractical to survey everyone and they full well know that people may not be truthful with their answers so they take appropriate measures to improve the statistics. First they over survey and ask cross checking questions. It is hard for people supplying false dat to keep their stories straight and they can be eliminated.

They also have excellent statistics as to what consists of a cross section of the target audience which is why they will ask you personal questions to set you up in the correct cohort. They already have information on you from the Bureau of Stats and the census as well as databases they buy from various data accumulators and retailers.

Lastly this is not about you personally. You may see yourself as the centre of the universe but nobody else does. You are just a sample and a very unlikely sample because unless the budget allows, a relatively small sample is all that is possible.

Obviously the larger the sample the more accurate the result but the survey companies are very good at getting useful results from smaller surveys. They qualify the results according to their calculated accuracy.

Deref 4:46 pm 21 Oct 15

dungfungus said :

The more I read about surveys and polls the more I remember that old political rule:
“Only hold a Royal Commission if you know what the outcome is going to be”

Exactly the same rule applies to surveys.

If samples of the question published in the Crimes are accurate, this has holes in it that are so big that you could drive a truck, without a permit, carrying a very high load through it without doing any damage whatsoever.

dungfungus 11:42 am 21 Oct 15

The more I read about surveys and polls the more I remember that old political rule:
“Only hold a Royal Commission if you know what the outcome is going to be”

Charlotte Harper 10:54 am 21 Oct 15

I’ll forward your comment onto them.

gooterz 9:11 am 21 Oct 15

Needs to be transparent but won’t say how the numbers are sourced.
I guess the question is, if I have a mobile number which is unlisted and on the do not call list. Do I have a equal chance of being polled as someone with 2 mobile numbers and a landline?

If a house is called is everyone at the house surveyed or do people I large houses have less chance to be selected. Being that you people whom live alone are guatenteed to be part of the survey if called and only one of the household if not.

Then what happens to non responses which of themselves is a bias. If someone refuses to take part they might also have a bias against it. Yet no factor is mentioned

dungfungus 7:55 am 21 Oct 15

If the company undertakes to be transparent in collecting data why doesn’t it post what it told you on their website?
There is a statement on the privacy section of their website referring to comments they receive not being given any confidentiality that is assured to respondents in the survey interviews.
Also, the way perceived age “skews” are addressed seems a bit dodgy.
As usual, the terminology “commercial in confidence” is used which is the usual response to mask some practice that could be considered unsavoury by general community standards. In this case it is the “geolocality enhanced” list of numbers supplied by phone companies.
You have done well Charlotte.

1 2 3

Related Articles

CBR Tweets

Sign up to our newsletter

Top
Copyright © 2017 Riot ACT Holdings Pty Ltd. All rights reserved.
www.the-riotact.com | www.b2bmagazine.com.au | www.thisiscanberra.com

Search across the site