The woes of government algorithms

From robo-debt to racism: what happens when government algorithms make the decisions.

Woes of government algorithms

Monika Sarder, Monash University

Algorithmic decision-making has enormous potential to do good. From identifying priority areas for first response after an earthquake hits, to identifying those at risk of COVID-19 within minutes, their application has proven hugely beneficial.

But things can go drastically wrong when decisions are trusted to algorithms without ensuring they adhere to established ethical norms. Two recent examples illustrate how government agencies are failing to automate fairness.

1. The algorithm doesn’t match reality
This problem arises when a one-size-fits-all rule is implemented in a complex environment.

The most recent devastating example was Australia’s Centrelink ‘robodebt’ debacle. In that case, welfare payments made on the basis of self-reported fortnightly income were cross-referenced against an estimated fortnightly income, taken as a simple average of annual earnings reported to the Australian Tax Office, and used to auto-generate debt notices without any further human scrutiny or explanation.

This assumption is at odds with how Australia’s highly casualised workforce is actually paid. For example, a graphic designer who was unable to find work for nine months of the financial year but earned $12,000 in the three months before June would have had an automated debt raised against her. This is despite no fraud having occurred, and this scenario constituting exactly the kind of hardship Centrelink is designed to address.

The scheme ultimately proved to be a disaster for the Australian government, which must now pay back an estimated $721 million in wrongly issued debts after the High Court ruled the scheme unlawful. More than 470,000 debts were wrongfully raised by the scheme, primarily against low income earners, causing significant distress.

2. Inputs embed racism
The stunning scenes of police violence in US cities have underscored the extent to which systemic racism influences law and order processes in the United States, from police patrols right through to sentencing. Black individuals are more likely to be stopped and searched, more likely to be arrested for low-level infractions, more likely to have prison time included in plea deals, and incur longer sentences for comparable crimes when they do go to trial.

This systemic racism has been repeated, more insidiously, in algorithmic processes. One example is COMPAS, a controversial ‘decision support’ system designed to help parole boards in the United States decide which prisoners to release early, by providing a probability score of their likelihood of reoffending.

Rather than rely on a simple decision rule, the algorithm used a range of inputs, including demographic and survey information, to derive a score. The algorithm did not use race as an explicit variable, but it did embed systemic racism by using variables that were shaped by police and judicial biases on the ground.

Applicants were asked a range of questions about their interactions with the justice system, such as the age they first came in contact with police, and whether family or friends had previously been incarcerated. This information was then used to derive their final ‘risk’ score.

As Cathy O'Neill put it in her book Weapons of Math Destruction: “It’s easy to imagine how inmates from a privileged background would answer one way and those from tough inner streets another.”

What is going wrong?
Using algorithms to make decisions isn’t inherently bad. But it can turn bad if the automated systems used by governments fail to incorporate the principles real humans use to make fair decisions.

People who design and implement these solutions need to focus not just on statistics and software design, but also ethics. Here’s how:

- consult those who are likely to be significantly affected by a new process before it is implemented, not after

- check for potential unfair bias at the process design phase

- ensure the underpinning rationale of the decisions is transparent, and the outcomes are relatively predictable

- make a human accountable for the integrity of decisions and their consequences

It would be ideal if the developers of social policy algorithms put these principles at the core of their work. But in the absence of accountability in the tech sector, numerous laws have been passed, or are being passed, to deal with the problem.

The European Union data protection law states that algorithmic decisions that have significant consequences for any person must involve a human review component. It also requires organisations to provide a transparent explanation of the logic used in algorithmic processes.

The US Congress, meanwhile, is considering a draft Algorithmic Accountability Act that would require institutions to consider “the risks that the automated decision system may result in or contribute to inaccurate, unfair, biased, or discriminatory decisions impacting consumers”.

Legislation is a solution, but it is not the best one. We need to develop and embed ethics and norms around decision-making into organisational practice. For this we need to boost the public’s data literacy, so they have the language to demand accountability from the tech giants to which we are all increasingly beholden.

A transparent and open approach is vital if we are to make the most of the technologies on offer in our data-rich world, while retaining our rights as citizens.The Conversation

Monika Sarder, Senior Strategic Analyst, Monash University

This article is republished from The Conversation under a Creative Commons licence. Read the original article.

If you enjoy our content, don’t keep it to yourself. Share our free eNews with your friends and encourage them to sign up.

RELATED ARTICLES





    COMMENTS

    To make a comment, please register or login
    tobymyers
    10th Jun 2020
    3:50pm
    You see the problem is from the start in that Centrelink don't pay enough in the first place.
    They make up their own cost of living, average wage and how they assess you.
    it
    s like an individual making up their own laws and disregarding the law of the land and then making it legal.
    I have just finished a survey on financial abuse and the more questions I answered the more I realised that Centrlink and the government are the greatest offenders of financial abuse .

    They decide how much to pay you , they then stop or reduce your payments if you don't comply to their rules and this is done regardless of what stress it places you under.
    Yes, they are the great abuses of people and it's about time people got off their lazy arses and did something about it.
    Jim
    10th Jun 2020
    4:04pm
    Does this article assume that all 470,000 infringements were wrong or that the system was unfair and that’s why all the debts are being repaid, taxpayers have long complained about rorts in the welfare system and governments of all persuasions have promised to do something about it, usually just before an election. The taxation PAYE taxpayers pay is based on earnings in a particular pay period, ie if a taxpayer works 12 hour shifts 7 days a week and their pay period is monthly, the amount of tax they pay could be astronomical, if they work one month on and one month off and their pay isn’t averaged over two months they would have paid an excess of tax, they would then have to wait until the end of the tax year to claim that money back, I think this is how the Robo system was meant to work, obviously major fail.
    Lookfar
    11th Jun 2020
    11:22am
    Jim, it was concluded that the debts were ILLEGAL so of course they were all wrong otherwise All the debts would not have to be repaid.

    There is no evidence to say that any number of people were actually rorting the system, except the debt collectors I suppose, and in any case the illegal robodebt, operating completely in la la land, would not have found them except by co-incidence, and it wouldn't have known that either..
    Tanker
    10th Jun 2020
    4:09pm
    There has been a tremendous rush into I.T. with a belief it can do everything. We have seen how that can come unstuck in a big way with aircraft as well as robodebt.
    It now appears that the production of automated cars on the road is doubtful because of the complexity of being fail safe.
    Jim
    10th Jun 2020
    4:24pm
    The article also decides to get into the racism debate, the buzz word at the moment seems to be systemic racism, it was quoted many times by the protesters in the black lives matter protest, one interviewer decided to question the protesters on the reasons they were protesting and whenever the said because of the systemic racism against black people, he asked them to explain what they meant, not one single person could give an explanation or an example of systemic racism, but they were sure it was happening? The interviewer was a person of colour who disagreed with the majority of reasons given for why people were protesting, instead he believed most of the white protesters were doing it to make themselves feel better about themselves without having any idea about the problems associated with the real issues facing certain groups in our community. Avi Yemini is the interviewer, I hope that’s the correct spelling.
    Lookfar
    11th Jun 2020
    10:46am
    Jim, just looking for the facts you base your assertions on and it seems to come down to one vague reference, " and if proof is needed, I suggest people read some of the comments by Avi Yemini a person of colour who disputes most of the arguments Put out by many of the activists,"
    This is not a fact nor an argument, indeed it doesn't even qualify as an opinion.

    I would sugeest that yor opinion that you have stated the facts is totally illusary so you have no right to claim that others should prove their facts.
    Buggsie
    10th Jun 2020
    4:52pm
    Let's not blame the algorithm for the robodebt problem. The real problem was the willingness of the federal ministers and others in parliament to proceed with a system that they knew was flawed and then to issue illegal debts to people who were vulnerable and quite unable to challenge these so called debts in the time frame allowed. The robodebt system was fraudulent and illegal from the start, nothing more than white collar crime committed by a government desperate for more money no matter what. No, it wasn't the algorithm, it was the willingness of politicians to use a system that they knew was unsound and illegal - in fact my local member, Dr David Gillespie, will not even concede that all white collar crime should be punished - I wonder why not?
    Horace Cope
    10th Jun 2020
    5:25pm
    I note that the current government is copping the blame for RoboDebt when all they did was continue with the system devised by Labor. RoboDebt was deemed to be illegal late last year, the recovery program was put on hold and, after investigation, refunds were announced. It is certain that refunds will be made to some who were overpaid and should be required to repay the overpayment.
    Lookfar
    11th Jun 2020
    10:59am
    Horace, the Liberal 'govt' has been bumbling along for over ten years, there was no RoboDebt 10 years ago, you should either go back and give your propaganda master the edge of your tongue, or prove your claim that the RoboDept as was found to be a criminal act in a properly constituted Court of Law, (for which the treasurer should be punished and severely, imho) which deemed the money had to be re-paid, and one would expect, within the same time frame demanded by the Robo Debt of the entirely innocent folk who were receiving the pension, etc. that they had been granted by the law.
    To my knowledge, this has not been done yet the time has passed, - just why on Earth do you support these criminals and their criminal actions?
    sainter
    11th Jun 2020
    11:09am
    KSS....for last time can you accept no for an answer....good-bye,if i'm blind I hope your not.....once again good-bye.
    sainter
    11th Jun 2020
    11:09am
    KSS....for last time can you accept no for an answer....good-bye,if i'm blind I hope your not.....once again good-bye.
    Horace Cope
    10th Jun 2020
    5:33pm
    The second part of the article refers to what is happening in America and there is no proof that the same things are happening in Australia. Aborigines make up about 3% of the population and about 28% of the prison population and if those raw figures are used it would be a very simple task to create a narrative suggesting that Aborigines are being targetted. The truth is that Aborigines are being incarcerated for crime that they have committed; bashing wives, molesting children and murder as well as repeated minor criminal matters.

    The Royal Commission in 1991 and a study late last year both dismissed the claims that more Aborigines die in custody than other races, even allowing for the previous percentages. I hesitate to write this because any comments refuting the lies put out by those who want changes made generally result in the immediate claim that the writer is a racist. It's easy to use the race card or the sexist card but they are usually played without any proof as the claim itself is deemed to be proof enough.
    Jim
    10th Jun 2020
    6:05pm
    I agree, and if proof is needed, I suggest people read some of the comments by Avi Yemini a person of colour who disputes most of the arguments Put out by many of the activists, but don’t expect anyone to agree, it doesn’t fit their dialogue.
    sainter
    10th Jun 2020
    6:48pm
    Rather than writing that post....go and tell a few aboriginal people about this Horace.
    sainter
    10th Jun 2020
    6:48pm
    Rather than writing that post....go and tell a few aboriginal people about this Horace.
    Jim
    10th Jun 2020
    7:27pm
    So sainter do you have any evidence That this isn’t the case, there are plenty of aboriginals that would disagree, there is an Aboriginal MP up north that doesn’t agree with the mainstream belief that Aboriginals are unfairly targeted, there are major issues with Aboriginal youth and much of this is being fuelled by white activists with their own agenda’s The thing that concerns me the most is that only white people can be racist, we won’t solve the racism problems on these pages, everyone has their own reasons for making the comments that they do and who they want to blame, the truth is that most blacks in the USA are murdered by other blacks, most white people are murdered by other whites. In Australia most Aboriginals are attacked and abused by other Aboriginals, we need to try and determine the cause, and not just blame someone who arrived here hundreds of years ago.
    sainter
    10th Jun 2020
    7:35pm
    I'll leave the evidence to experts like you Jim you seem to know it all,it doesn't take much to work out what a racist is, and i don't need someone like yourself to tell me Jim,you just keep showing us all your figures Jim.
    sainter
    10th Jun 2020
    7:35pm
    I'll leave the evidence to experts like you Jim you seem to know it all,it doesn't take much to work out what a racist is, and i don't need someone like yourself to tell me Jim,you just keep showing us all your figures Jim.
    Jim
    10th Jun 2020
    8:31pm
    There is only one person here who thinks they are an expert on the aboriginal problem, I haven’t tried to denigrate anything you have said, I have just stated the facts, you have provided no evidence other than a derogative comment, which says more about you than it does about me, if you can show any evidence from my comments that suggests I think I am an expert on the aboriginal problem, I would like you to highlight that evidence, you don’t know me so how can you make the comment that I claim any expertise on anything, I certainly don’t make that comment, as for me telling you what a racist is I haven’t done that either, the comment I make is that there are racist in all walks of life. I show the figures to prove there is a different side to the story, by prove I mean the information that is available from different sources, is any of it true I don’t know, do you know if any of the information that’s available to you is true? Unless you are Aboriginal living in an Aboriginal community then you wouldn’t know if your information is correct either, but feel free to vent your offence against someone who has a different opinion, I think we call that bigotry don’t we?
    sainter
    11th Jun 2020
    6:48am
    Don't have to show you any evidence Jim read what you write mate,you've got right to your opinion or opinions Jim....have a great life...and don't refer to me as a bigot,iv'e gone through racial slurs because of my families background....and Jim i'm not aboriginal or black.
    sainter
    11th Jun 2020
    6:48am
    Don't have to show you any evidence Jim read what you write mate,you've got right to your opinion or opinions Jim....have a great life...and don't refer to me as a bigot,iv'e gone through racial slurs because of my families background....and Jim i'm not aboriginal or black.
    Tanker
    11th Jun 2020
    7:55am
    Avi Yemini is a far right "foghorn" in that he conducts interviews deliberately designed to cast doubt on issues that the far right see as a chance to drive a wedge into society. He does a somewhat dubious linkage with Fascist groups in the U.K.
    Jim
    11th Jun 2020
    8:12am
    My apologies sainter, it was never my intention to accuse you of bigotry. Tanker I was not aware of any links to the far right that you say Avi Yemini is involved with, I saw his interviews with over a dozen people who clearly hadn’t a clue regarding the protest, now he might be in there stirring the pot, but there is also an element of left extremists doing the same thing, if we genuinely took out the political agenda’s we might get somewhere, I think it’s true to say we have a problem with our indigenous people, but I think it’s more complex than just a racist issue, we have been trying to address this issue for as long as I can remember, if anything we seem to be going backwards. We all have a right to our opinions, but we should always be respectful of other people’s opinions.
    KSS
    11th Jun 2020
    8:45am
    "i'm not aboriginal or black" and yet you choose to speak on their behalf Sainter. How can you possibly know what that is like?

    Sainter I would refer you to Jacinta Nampijinpa Price, Aboriginal woman and councellor from Alice Springs, and Nyunggai Warren Mundine Aboriginal man who have both written and spoken at length on real Aboriginal issues. Most recently yesterday Ms Price wrote about the BLM protests which she considers misplaced and factually plain wrong! Mr Mundine labelling the protests as 'irresponsible'.

    The truth is that Aboriginal people are NOT more likely than any other Australian to die in prison. Nor are they killed at the hand of the police. Do the research and you will see these are facts, not opinion. Oh that's tight, you don't actually want to see the evidence because you are so sure you are right.

    It is true that Abroiginal people make up about 28% of the prison population and only 3% of the general population. However, they are imprisoned for crimes mostly against other Aboriginal people. Most Aboriginal deaths from violence are at the hand of other Aboriginal people and people they know from the same community. This is not to justify these crimes or belittle them in any way. It is truly shocking. BUT it is NOT racist to state the truth.

    And just for good measure, a more than cursory look at figures in the USA show similar outcomes. Most black people are victims of crime by other black people, not the (mainly) white police force. Do the police overstep their authority sometimes? Of course and they should feel the full weight of the law when they do. But to label the entire USA police force as racist is plainly wrong. And further, the issues in the USA are NOT comparable with the issues in Australia. There is quite enough here to deal with without the need to import irrelevant misrepresentations from overseas.

    And finally, three years ago a white woman was shot dead by a black policeman in Minneapolis. Her crime? Reporting what she thought was a sexual assault on a woman behind her house and going outside to meet the police. It took two years to bring that policeman, Mr Noor, to court. Where were the protests then at police violence? This in the same Minneapolis that witnessed the violence against a black man who also died at the hand of a policeman two weeks ago that sparked the world wide riots. That policeman has faced court after just a few days. The difference, this time the policeman was white!

    The point is, if people of colour do not want to be judged by the colour of their skin (and rightly so), then white people should also not be judged by the colour of their skin either.
    The very use of the perjorative "white privilege" is itself racist! No-one, not even you, needs to apologise for their skin colour.
    sainter
    11th Jun 2020
    9:01am
    Apology accepted Jim,we all including myself get a bit hot over issues,in the end i'm glad we can accept each others opinions....no hard feelings from me mate.....have a good one and I really mean that...thanks all the best.
    sainter
    11th Jun 2020
    9:01am
    Apology accepted Jim,we all including myself get a bit hot over issues,in the end i'm glad we can accept each others opinions....no hard feelings from me mate.....have a good one and I really mean that...thanks all the best.
    sainter
    11th Jun 2020
    9:05am
    KSS...I've read some of your post previously....nothing has changed except I will avoid them in future.
    sainter
    11th Jun 2020
    9:05am
    KSS...I've read some of your post previously....nothing has changed except I will avoid them in future.
    Jim
    11th Jun 2020
    9:48am
    Cheers sainter, I agree
    KSS
    11th Jun 2020
    10:23am
    Sainter, there's none so blind as those who will not see! Ignore my posts, that is your perogative. But the truth remains.
    Lookfar
    11th Jun 2020
    11:13am
    DSS, on a point of order you admit that a much higher number of black deaths in prison in proportion to white deaths in prison is a correct statistic, but then you claim, with no evidence at all that that is somehow cancelled by your assertion 'that they were mainly black doing the killing? is that what you meant, the people who died in prison had mainly killed black people? - what, before they were put in prison?
    So the black deaths are caused by those dying having killed other black folk before the went to prison? - Have you any proof? - trials, sentences, etc? or are you maintaining that those, often in solitary confinement, somehow walked through the walls and killed other black people in high security prisons?
    I think Fact Check would have a fine old time with you, but at least you may be introduced to Reality.
    KSS
    11th Jun 2020
    1:20pm
    I assume you are referring to my comment Lookfar, but if so you have it wrong. There is no evidence that more black people die in prison than white people nor that black people die as a result of police violence more frequently than white people. The last Royal Commissison was very clear about that, likewise a report just last year also backed it up.

    "Of the 2,608 people who have died in custody since 1979–80, 500 were Indigenous and 2,104 were of non-Indigenous background." https://aic.gov.au/media-centre/media-releases-and-statements/latest-deaths-custody-data-now-available-crime-statistics-australia

    "And 58% of Indigenous deaths in custody were due to natural causes, hanging 32%, 5% drugs or alcohol, 4% external trauma. For each year from 1991–92 to 2002–03,
    the leading cause of death among Indigenous prisoners was either natural causes or hanging. For each year from 2003–04 to 2015–16, deaths due to natural causes surpassed hanging deaths." https://aic.gov.au/publications/sb/sb17

    Yes Indigenous people are over represented in prisons but they are not there because of police violence or racism. There are there because they committed crimes. And most of the crimes committed were violence against other Aboriginal people.

    The real issues to focus on should be the underlying reasons for the high crime rate in Aboriginal populations not some confected discourse on non-existant endemic police or general population racism.
    Lookfar
    11th Jun 2020
    11:35am
    My first experience of systemic racism in Australia, was during the anti Vietnam war protests, where the police would walk into a crowd of demonstrators and just arrest the few black folks in the crowd.
    Have things changed now?

    On the white side, whilst living in Kuranda it was habitual for the white folk to blame the black folk if anything was stolen, and my swish pushbike was stolen, the police couldn't help, and it was just chance I saw my quite distinctive bike, although painted grey, in a group of white boys in town, - one of which leapt onto it and tore straight across the road and down a big hill, lucky he wasn't killed, although he did get away.

    No doubt at all that he was a high-school age white boy though, so I keep my own counsel when now I hear white folk mouthing off blaming the blacks for everything, it is usually just racism.
    greenie
    12th Jun 2020
    6:11pm
    Don't see how the USA example is relevant to us here. Why mention it. A waste of my time reading it. Go away!
    Lookfar
    12th Jun 2020
    6:16pm
    Greenie, please explain or you should go away.


    Join YOURLifeChoices, it’s free

    • Receive our daily enewsletter
    • Enter competitions
    • Comment on articles