Wednesday, September 20, 2017

Recent Polling On The Same-Sex Marriage Postal Survey

The national ABS postal "survey" on whether the law should be changed to allow same-sex marriage in Australia is now in its second week.  A number of pieces of polling have been published or alluded to since my last general polling update, but what do they really tell us about the outcome and how reliable are they?  At this stage there is still much that we do not know.  It is too early to be certain Yes has it in the bag, but the widespread narrative that support for Yes is crashing rapidly and that this is another Trump or Brexit coming is so far not that well supported by the evidence.

Public Polling: Ipsos

Firstly, the public polling.  Last week saw an Ipsos poll which buoyed some worried Yes supporters with a 70-26 Yes response, one of the highest Yes votes ever recorded in a poll in Australia.  Indeed, as far as I'm aware, this score has only been exceeded in a few commissioned polls and one Morgan-SMS (a suspect polling method) which did not use an undecided option.  The Ipsos also found a 70% Yes response among the 65% of voters who rated themselves as certain to vote, and found a gender gap with 72% of women and 59% of men saying they were certain to vote.

However, I am treating these Ipsos figures with a lot of caution.  Firstly we know that Ipsos greatly overestimates the Green vote compared to other pollsters and compared to election results, and Green voters very strongly support same-sex marriage.  Although that's probably only worth a point or two on the headline figure for same-sex marriage, it could be a concern if whatever causes Ipsos to over-poll the Green vote also might cause them to over-poll the left end of the Liberal base.  Some evidence for the latter concern might be suspected in the perpetually strong results for Malcolm Turnbull in Ipsos compared to other polls.

The second issue is whether Ipsos, as the only national pollster still using exclusively live-interview phone polling, is affected by "social desirability bias".  When talking to another person on the phone, voters may be more reluctant to say they oppose same-sex marriage, since they may be concerned about offending the interviewer.  When clicking boxes on a computer survey, or pressing numbers on a robo-poll, these issues don't arise.  Some overseas studies (mainly American) have found live phone polling more prone to desirability bias than other polling methods, while some haven't, and some have even challenged whether it's a thing in polls at all, but I don't recommend assuming that it isn't.  A spectacular difference that may have been down to this was that between the phone polls and the online polls in the Brexit vote.  The phone polls had large leads for Yes until very late in the piece while the online polls always had a close contest.

In the Australian context, we don't have enough pollsters of each type to really draw a line between phone polls and other polls.  But I've noticed over time that the live polls (Ipsos phone, old Newspoll phone, old Nielsen) tend to get higher-end results for Yes, while one particular robopoll (ReachTEL) tends to get much closer results than other polls in general.  The closer results in ReachTEL occur not only nationally but also locally.  This was especially pronounced in the early days of ReachTEL when the pollster used to get awfully bad results for Yes (like 43.0% in 2011) but has remained to some degree up til the present.

Public Polling: Essential

This week's Essential had a number of interesting features.  See the main poll question and also the question about likelihood of voting.

Firstly, this Essential finds a 55-34 headline result in favour of changing the law to allow same-sex marriage, which is (by a small amount) the highest No response I can find in a published poll that includes a don't know option since 2009, excluding the early ReachTEL mentioned above.  The Essential continues the trend of all previous public polling of having voters who say they will definitely vote more pro-Yes than those who say they probably won't. 

As Peter Brent pointed out on Twitter, this week's Essential sample was a strong one for the Coalition compared to other recent samples, so it's plausible that random error has made this result slightly closer than it would have been.

An unusual aspect of the plebi-survey is that it allows for pollsters to conduct exit polling of a sort while people are still responding.  Thus this particular poll finds 9% of respondents have already voted, and of these 59% say they support changing the law, 37% say they don't, and 5% say they don't know.

The 5% who say they don't know are rather baffling, since after all they have already answered the same question in the survey.  For the Yes side, a concern would be that these may be voters who heeded the "if you don't know, vote No" rubbish from the No campaign.  (I call it rubbish because if you don't know, you don't even have to vote at all.)  The question doesn't yet ask how did you voteThere may even be some people who support changing the law but voted No for some bizarre reason, though their numbers are probably very small if so.

Now 59-37 may still sound pretty good for Yes (assuming Yes is happy to win by any reasonable margin), but there are again reasons for caution.  Firstly the sample size: the margin of error on 9% of a one-week Essential sample (less than 100 voters) would be about 10% if it was treated as a random sample, ignoring the increases in effective error margin that come from the use of scaling.  (There is also the point that panel polls are self-selecting samples of a non-random selection of the population anyway, so applying "margin of error" to them is especially difficult).

More importantly, exit polling of this kind will be stratified, both by time of receipt and by time taken to respond.  The Yes and No votes are unlikely to be constant in time through the survey.  Voters in different areas will receive their survey forms at different rates.  Those in remote areas or whose forms have to be redirected from a different address may receive them more slowly.  When voters get their surveys, those with strong feelings are more likely to vote right away, while those who care little about the issue may defer a decision on voting, and those who are genuinely unsure may take longer to think about it.  Furthermore, campaigning through the survey period will influence vote behaviour.  The first 9% might be representative of the final outcome, but it's not something I'd be at all sure of.

Public Polling: YouGov

YouGov (another online pollster) this week has a 59:33 lead for Yes with 9% unsure.  This is unchanged from a month ago except that in that case 8% were unsure (so the only difference is the unsure figure after rounding.)  So that's one poll not backing the idea that the Yes vote is declining.

Internal Polls

Some internal polls conducted for the two campaigns have been reported.

Two online polls by Newgate Research for the Yes campaign have been reported, the first of which showed 63% for Yes and the second of which showed Yes leading 58.4-31.4.  The second was reported as having Yes down six points and No up two, which suggests something in the various reporting of these polls is not exactly accurate!

Newgate Research is a relatively little-known pollster and has not released any voting intention polling prior to elections based on which its accuracy can be gauged.  It's Senior Director Jim Reed is formerly of Crosby/Textor, the far better known internal pollsters for the Liberal Party.

The reporting of the second Newgate poll was picked by several people as a classic instance of that poll reporting trope, the internal polling scare story.  For a rather modest shift off a modest sample size (800) we got that support "has crashed", that "only" two-thirds of voters intended to take part, that this would be a "very low" turnout, "alarmingly" so for Yes campaigners, and so on.  The reporting said the poll was released as a "wake-up call", which rather implies that had its results not supported such a call it wouldn't have been released, ie selective release.  As evidence that Yes was at risk of going down in a screaming heap, a 27-point lead was far from convincing!  However, the idea that No has been making at least modest gains is now supported by the Essential result noted above.

The Newgate poll report noted with concern that young voters were less confident of voting, but Essential has found this too and yet still found that Yes voters are more likely to vote than No voters.  This is apparently explained by Yes supporters saying they're more likely to vote than No supporters across every demographic.

There are some strategic concerns about whether encouraging this style of reporting through the use of internal polls to scare people into not being complacent was a good idea.  Overhyped reports of support for same-sex marriage "crashing" may feed into a bandwagon effect for the No campaign.

The No campaign's Lyle Shelton claims to have internal polling showing a drop in the Yes vote from 65% to 58% , on which basis he claims that Yes has shed "over 1 million votes in ten days".  The latter claim is false as he is forgetting that not everyone will vote.  In any case, no details of pollster or method have been provided.

Another report from the No campaign says that they have support among female voters dropping from 67% to 60% in ten days, and male voters 54% to 49% in the same time, with supposedly a large drop in support among Gen X voters and negligible change among young voters.  Again, no details of pollster or of method and again, inconsistency in the different claims about the polling.

ReachTEL has been conducting widespread internal polling for itself - these results won't be released and are for the purposes of prediction-testing.  (The article is fascinating but raises some questions about how the pollster has "about 100 different variables on each individual" and whether this is problematic in privacy terms.)

 A more contentious robocall has emerged conducted by American firm WPA Intelligence.  The poll asks respondents their views about same-sex marriage, then presents them with two statements, one pro-same-sex-marriage and one against it.  Then it asks them their view about same-sex marriage again. This has been claimed to be a "push-poll" but I think it is more likely to be some kind of message-testing exercise, albeit one that should be a lot more upfront about who it is being conducted for.   For a detailed discussion about the differences between push-polling, skew-polling and message-testing see this piece from the 2014 Tasmanian state election.

Alleged Internal Poll (Unverified)

Wirrah Award For Something Fishy (Not Entirely Clear What) 

An especially dubious internal polling report came from one of the usual suspects on this issue, Miranda Devine.  Devine reported on "Exclusive polling by the Coalition for Marriage" in which 5% of 4000 adults surveyed identified as "LGBTIQ" and of these 93% supported same-sex marriage.  So far so unproblematic, although it probably won't stop said so-called Coalition from continuing to claim that only the rights of one point something percent are affected.  However Devine then, apparently in reference to this LGBTIQ cohort, claims:

A staggering 92 per cent would vote “No” or boycott the ballot if the proposed change to the Marriage Act “has not been thought through properly in terms of all of its consequences for the majority of Australians”.

While I don't expect Devine or other such op-ed authors to know better, this simply doesn't pass the smell test.  Far more than 8% of such voters would be immune to such blatant skew-polling and either click yes anyway or else abandon the survey in disgust.  And ditto for these further claims:

And 81 per cent would vote “No” or not vote if it turned out that the “Yes” campaign was “being run and controlled by left-wing activists with a hidden agenda that goes well beyond changes to the Marriage Act”. [..]

The survey also showed that more than three-quarters of LGBTIQ respondents would vote “No” or not vote if changes to the Marriage Act triggered “negative consequences for the majority of Australians, including restrictions on free speech, and penalties for acting according to one’s beliefs about marriage”.

Almost two thirds say they would vote “No” if same-sex marriage resulted in “forced exposure of young children to radical sex education content without parental consent”. A further 12 per cent would boycott the ballot.

And one third would change their vote to “No” or not vote if the “Yes” campaign were “nasty”, understanding that they would bear the backlash.

In other words, LGBTIQ Australians have serious reservations about the consequences of changing the definition of marriage.

For us to even consider taking such implausible claims seriously, these claims should have been backed by details such as the pollster conducting the polling, the polling methods, and the full list of questions.  But even if these results are genuine and conducted by a reputable pollster - and not completely concocted or grossly incompetent as they sound - it is obvious that the poll would have engaged in severe skew-polling.

Beware Cherry-Picking

Some media sources that support same-sex marriage have nonetheless been keen to play up the narrative that the Yes vote is crashing and the Yes campaign is terrible.  Partly this makes for a more interesting narrative (there is often more interest in contests perceived to be close or unpredictable) and partly it helps some of them to reach out to No campaigners by saying "well yes I support same-sex marriage but the Yes campaign are doing it all wrong".

A good example was a tweet from Paul Murray that drew a line through Morgan 76% in March 2016, Newspoll 63% in August, Essential 55% now and asked "Time to worry?" But the Morgan result, as already noted, was a dodgy yes/no SMS poll with no undecided option (practically an opt-in); other polls from 2015 and 2016 had an average Yes response of about 62%.  Also, Newspoll has generally had slightly higher Yes votes than Essential, mainly because it tends to have lower don't-know votes.  And why include an ancient Morgan and not a modern Ipsos?

Intentionally or otherwise, anyone can pick points from a sample that support any conclusion one likes - that support for same-sex marriage is plummeting, rising or staying still.  But it is better to look at a full range of polls.  An incomplete list may be found on Wikipedia, but be aware that some are commissioned and some are not.

If The Gap Is Narrowing, Why?

The above polling is not really conclusive on whether the gap between Yes and No has narrowed in recent weeks.  On an aggregate of public polls weighted for house effect it is, but only very slightly.  Two of the three public polls do not find narrowing.  Reported internal polls of both sides find narrowing, but both sides perceive themselves as having a reason to claim this: the Yes campaign to discourage complacency and the No campaign to encourage momentum and media attention and a belief that voting No isn't futile.

If we assume there has been a significant drop in the Yes vote, some would blame it on the antics of the more indisciplined or extreme Yes supporters.  That would be a part of the story (if there is one) but my view is that if someone says "I was going to vote Yes but then those nasty Yes supporters were meanies to the poor No supporters, so I'm voting no because they might be nasty to me too" there is a fairly high chance that that person is concern-trolling.  After all I am yet to see anyone at all saying they were going to vote No but switched to Yes because Kevin Rudd's godson was bashed, because actual fascists put up crackpot homophobic posters or because a church refused to marry a couple who had expressed support on Facebook for same-sex marriage.

Referendum-type "debates" often, but not always, see some level of decline in support for the pro-change position.  They tend to, as this one has done, become arguments about and proxies for everything but the issue in question.

They are also prone to objections of the "yes, but not like that" form, and can be said to be "built to fail".  If the legislative model is specified then some people who support the concept may not like the model.  If the legislative model isn't specified (as in this case) then some people will say that since they don't know what they're voting for they cannot vote for it.  Most people seem to either realise it is actually and intentionally an in-principle question, or else not care about the precise details of religious exemptions.

Is This Trump/Brexit/the Republic Again?

There have been many attempts to argue that the postal survey will return an upset No vote, using the election of Donald Trump, the win for Leave in Brexit, and the 1999 defeat of the Republic referendum as precedents.

At this stage these attempts are not convincing.  The polling misses in final polls in the cases of Trump and Brexit were not massive.  National final polls had Hillary Clinton beating Donald Trump by an average of about four points in the popular vote, and she beat him by two.  The Brexit miss was worse - polls had Remain winning by about three points, and it lost by 3.8.  But neither provides a precedent for the loser to win while trailing by over 20 points (and an average exceeding 30 points on a yes-no basis) in every published poll.

In the case of the Republic proposal in 1999, support for the generic proposal that Australia should be a republic led opposition by around 15 points in published polls.  But the split in pro-republic forces between direct and indirect election models for the President was a massive one that led to very large numbers of republicans voting no in the (pious as it turned out) hope of bringing on a new vote on a direct-election model within a decade.  As a result the defeat of the republic referendum by 10 points is not a useful precedent either.

There is not, at this stage, a substantial faction of Yes supporters pushing to defeat the postal vote in order to pass a version with specific religious safeguards included or excluded later. In general the complaints about lack of clarity regarding exemptions are coming from those who will be voting No anyway, and they are making these complaints in a classic game of FUD to try to pry "uncertains" off the fence.

There is also not, so far as I can tell, a significant boycott from voters who would otherwise vote Yes.  There are some same-sex attracted people who are affected by the vote to such an extent that they do not feel they can participate, but their numbers will be behind the decimal point in the overall survey result.  Calls for a more widespread boycott from unaffected voters appear to have dissipated - indeed I've been tracking these on social media where they have slowed to an infrequent drip and are mostly made by insincere troublemakers.

So the idea that Yes will fail, given the polling, relies on at least one of these:

* The view that the unfolding "debate" (if you can call it that, it's mostly just an exchange of opposing views that was all done to death years ago) will radically transform opinion leading to vast numbers of voters changing their intended answers, probably in the next two weeks, or

* The view that there is some kind of systematic and massive failure in the polling of the question.  This would arise, for instance, from people who said they were certain to respond not actually responding, from social desirability bias contaminating all the different forms of polling, or from all the pollsters oversampling politically engaged voters on a scale that made the UK 2015 flop look tame.  Another possibility is that pollsters turn out to be asking the wrong question and that there is a large difference between how voters respond to the poll question when asked it in a poll, and how they respond to it on the survey form.  (Someone might say they agree the law should be changed but vote No anyway, for example.)

All of these seem to be big stretches but we should bear in mind that Australian pollsters greatly lack experience in how to sample voluntary voting, and it is much harder than sampling for compulsory.  Some other possible sources of polling failure might include the unusual nature of the "voting" process and any impacts of progressive exit polling on voter behaviour.  The Yes campaign will need to guard against supporters assuming that Yes will win based on early exit polling and therefore not bothering to vote.  Enthusiasm should be built for the idea of not just winning but if trying to win by as much as possible.

I did note in the previous article that there was a large failure in polling on this question in Ireland, with polls overestimating Yes on a two-answer basis by nine points and hence the margin by eighteen.  If this happened in Australia, on current polling Yes would still get home, by a margin of something like 57-43. 

On current polling I can't rule out that some combination of campaign shift, multiple polling errors and maybe vandalism could get No over the line.  But it would take more than was seen with Trump, Brexit and 1999 to do it.

An Alternative Approach

An alternative approach to prediction may be seen at, which tracks search activity for "vote yes" and "vote no" via Google trends and on this basis has so far been consistently showing No winning by a large margin.  The author used similar methods in a Greek referendum but the claim that they nailed the Irish same-sex marriage debate appears to be retrospective, rather than one where there was a public predictive test in advance of the vote.  It will be interesting to see how this goes, but it is not at all clear to me why "vote yes" should be a commoner search term than "vote no" in a country where the debate has been going on for many years and a great many Australian voters would not Google anything prior to answering.  It may be a pointer to a tendency of undecided voters to investigate or switch to No that may narrow the margin, or there may be something else going on with it.  Comments from anyone with expertise in this area welcome.


It may also be of interest to note that one bookmaker (Sportsbet) was at $3 for No in late August, dipped down to $2.25 and is now back at $3, while William Hill which was at $2.40 at one point is now at $3.75.  As has been noted here before, betting markets are not great predictors.


Mr Kenny thinks that posting pictures of your vote on Twitter is virtue-signalling so I thought I should be doubly virtuous and do it on here as well.  

Tuesday, September 12, 2017

A Free Speech Problem With Marriage Law Survey Safeguards

Advance Summary

1. This article raises concerns about specific "hate speech" prohibitions in the Government's Marriage Law Survey (Additional Safeguards) Bill 2017.

2. This article argues that Sections 15(1)(a) and (b) place unreasonable constraints on free speech by making political opinions attributes that are protected from "vilification", contrary to the normal practice of anti-vilification laws.

3.The ability to express strong criticism of people who present offensive or unfactual opinions serves as an important deterrent against expressing such opinions in the first place.

4. Many aspects of the proposed Sections and the limited exemptions available are insufficiently clear to a lay reader and involve a novel area of Australian anti-discrimination law.

5. Sections 15(1)(a) and (b) should be amended so that they apply only to intimidation and threats and not to "vilification". 

6. If this does not occur, then the debate surrounding the postal survey is not an adequately and clearly free and fair environment for the frank exchange of opinions and criticism.

Monday, September 11, 2017

Response From ABS to Marriage Law Postal Survey Questions

On 16 August I sent the ABS a list of fourteen questions regarding the conduct of the Marriage Law Postal Survey, in particular regarding count quality assurance issues.  Some of the questions were answered in subsequent public debate.  The response below was received today, September 11, from Michael Wilson of ABS and is reproduced in full.  My questions as sent are in italics.  My thanks to the ABS for their detailed responses at this busy time.

I have added some comments of my own below the responses, and may add more later.

Saturday, September 9, 2017

Thylacine: Specimen Or It Didn't Happen

This week a group of Tasmanians (press conference hereclaimed to have seen and to have footage of a living thylacine, a species which has not been confirmed to exist since the last known specimen died in captivity on September 7, 1936.    No zoologist has yet accepted this extremely blurry footage as being of a thylacine, and many wildlife observers consider it is very likely to be a spotted-tailed quoll.

You can see a longer video here.  It contains unconvincing (compare actual accounts) "barking", the video above, something unidentifable nosing the camera, and a bunch of Where's Wally pics in which you can just make out what might be the eyes of something if you try very hard.  When I slow down the main video frame by frame I can see blurry paler patches on the animal consistent with the spotting on a spotted-tailed quoll.  The most interesting thing about the videos is actually the large number of lyrebirds (introduced to Tasmania) that they show.

I thought it might be of some interest to someone out there to outline my position regarding this poor animal the thylacine, and the intermittent circus of alleged "sightings", "photos" and "videos" surrounding it.  As usual, I am expressing my own view and not necessarily the view of any organisation I belong to or any employer I from time to time work for.

Thursday, August 31, 2017

Not-A-Poll: Best Prime Minister Of The Last 45 Years

Round 1

Now and then we see newspaper polls rating the best PMs of the last few decades, or in the case of one Essential poll last week, the best government of the last ten years.  John Howard is a persistent "winner" of the first class of polling, but I've always believed he has an unfair advantage.  He tends to get a very high share of the Liberal vote with other Liberal PMs hardly getting any, while the Labor vote tends to be split up more between Hawke, Keating, Whitlam, Rudd and Gillard.  For that reason it's not clear whether Howard would beat all the Labor PMs on a head-to-head basis, although it looks like he probably would.

I've decided to run a similar Not-A-Poll exercise here in the sidebar just for fun over a period of several months.  The basic rule is that we keep going eliminating one PM at a time (perhaps more) until someone has over 50% at the end of a month.  The more complicated rules are:

Thursday, August 24, 2017

ReachTEL: Bob Brown Foundation Peddles A Poll Porky

ReachTEL (commissioned) Liberal 41.2 Labor 33.3 Green 13.1 PHON 4.4 Ind/Other 8.0
Interpretation Liberal 41.2 Labor 37.1 Green 10.4 Other 11.3
Most likely seat result based on this poll 12-10-2 plus one seat undecided between the three main parties
Disclaimer: Polls are snapshots not predictions.

I may get taken off their media mailing list for saying it, but the Bob Brown Foundation have released a grossly deceptive claim concerning their current ReachTEL poll to the Tasmanian press.  Question two of their poll and its results are as follows:

The question design is a bit odd, but I think it is OK.  The results are 38.3% support, 44.7% oppose and 17.1% don't know/not sure.

Tuesday, August 22, 2017

Poll Roundup: Newspoll Moves! It Is Alive! (includes same-sex marriage polling)

2PP: 53.5 to Labor (+0.3 since last week)
Labor would win an election "held now" with a large majority

Time for another roundup of the state of play in federal polling.  This week's exciting development was that Newspoll, after a record six consecutive 2PPs of 53-47 to Labor, finally moved by a whole point to 54-46!  I am especially excited by this because had this week's result been another 53-47, I was going to lead off with a mock Death Certificate, and now I do not have to.

This briefly took my aggregate to an equal term high 53.8 to ALP (I tweeted this as 53.9 but later found a very small error) but then Essential moved back a point the other way after two 54s, and the primaries in YouGov didn't do a lot, so I currently have it at 53.5 to ALP. That was the Coalition's winning lead in the 2013 election, and also just shy of what Labor had when Tony Abbott was removed.  

The strange 2PP results of YouGov continue to baffle - this week it has the Coalition with a 51-49 lead off primaries that would normally imply something like 53.1% to Labor.  I discussed this issue last time, and the average difference between YouGov's 2PPs and the last-election 2PPs for their primaries is now running at a massive 2.9 points.  As noted last time, it's possible that the current last-election 2PPs are overstating Labor's lead, especially because of One Nation issues, but it's highly unlikely that they're doing so by three points.  Much more detail on the YouGov mystery from William Bowe here.