Just because AI can do it doesn't mean it should
Good AI use saves hours. Bad AI use costs customers, money and trust. A simple judgment test for what to automate, what to assist and what to keep human.
AI can write a reply to an angry customer in five seconds. It can also make that customer angrier in five seconds.
That’s the whole topic in one line. The tools are good enough now that almost any task can be handed to AI. The skill that matters is judgment: knowing which jobs to hand over completely, which to share, and which to keep.
Three times automation went wrong
These aren’t small-business stories, but the lessons scale down perfectly.
A chatbot made a promise the business had to keep. In 2024, Air Canada’s website chatbot told a grieving customer he could claim a bereavement discount after booking. That wasn’t the airline’s policy. Air Canada argued it wasn’t responsible for what its chatbot said. A Canadian tribunal disagreed: the chatbot was part of the airline’s website, so the airline was responsible for everything on it. The lesson: whatever your AI tells customers, you said.
A report went out with references nobody checked. In 2025, Deloitte refunded part of a $440,000 report for the federal Department of Employment and Workplace Relations. The original version included references to reports that didn’t exist and a made-up quote from a court judgment. The corrected version disclosed that a generative AI tool had been used. The lesson: AI drafts confidently. Someone has to check before it carries your name.
An automated system made decisions about people. Robodebt wasn’t AI, but it’s Australia’s clearest lesson in automating a decision that needed human judgment. It used averaged income figures to raise welfare debts automatically and asked people to prove they didn’t owe the money. The Royal Commission called it “a crude and cruel mechanism, neither fair nor legal”. A class action settlement covered about 430,000 debts. The lesson: when a decision affects someone’s money or rights, a person needs to own it.
The judgment test: five questions
Before you automate anything, ask:
- Is it the same every time? Rules-based jobs, like moving an invoice into a folder, are safe to automate. Jobs that need reading between the lines aren’t.
- What happens if it’s wrong, and can you undo it? A misfiled document is easy to fix. A wrong price sent to a customer, or an email sent to the wrong person, isn’t.
- Who sees the result? If a customer, a supplier or the public will read it, a person should approve it first.
- Does it affect someone’s money, job, rights or wellbeing? Hiring, credit, debts, complaints and anything to do with health or legal matters. Keep a person deciding.
- Would you be comfortable if they knew AI did it? If the honest answer is no, that’s your answer.
Automate, assist or keep human
Once you’ve asked the questions, most jobs fall into one of three groups.
| What it means | Examples | |
|---|---|---|
| Automate | It runs on its own, and someone checks the results now and then | Appointment reminders, filing documents, logging new leads in your CRM, moving data between systems, alerts to your team |
| Assist | AI drafts or prepares, and a person approves before anything happens | Enquiry replies, quote drafts, review responses, proposals, meeting summaries, payment reminders |
| Keep human | A person decides, and AI can summarise or prepare background at most | Complaints and upset customers, hiring and firing, discounts and pricing exceptions, credit or debt decisions, anything legal, medical or financial advice |
The middle group is the biggest, and it’s where most of the time savings are. You don’t have to choose between doing everything yourself and handing it all over.
Good use vs bad use
The same tool can be used well or badly. The difference is almost always where the person sits.
- Good: AI drafts a reply to every enquiry, and you approve it on your phone. Bad: AI replies to complaints on its own.
- Good: AI summarises a pile of job applications against your criteria, and you read the shortlist and decide. Bad: AI rejects applicants automatically, without anyone looking.
- Good: AI drafts a reply to each Google review for you to check. Bad: AI writes reviews. The ACCC is clear that businesses must not create fake or misleading reviews, or arrange for others to.
- Good: AI pulls the numbers together for your monthly report, and you write what they mean. Bad: AI writes a report that nobody reads properly before it goes out.
- Good: A website assistant answers from your own documents and hands over to a person when it isn’t sure. Bad: A chatbot that guesses your policies.
- Good: Polite payment reminders go out automatically, and anything firmer waits for your approval. Bad: Penalties or debt notices sent without anyone checking the account.
The rules are catching up
From 10 December 2026, businesses covered by the Privacy Act must say in their privacy policy if they use computer programs to make decisions, or do things substantially and directly related to making them, that could significantly affect someone’s rights or interests using their personal information. The OAIC has published guidance on what that means.
The National AI Centre’s Guidance for AI Adoption sets out six essential practices for businesses. Two of them sum up this whole article: decide who is accountable, and maintain human control.
This is a plain-English overview, not legal advice. If you’re unsure whether the Privacy Act applies to you, check the OAIC or ask your adviser.
Judgment is a setting you can change over time
How we approach this with clients:
- Approvals start on. Everything customers see is approved by a person at first.
- Trust is earned. Approvals are relaxed only on jobs that have run well for weeks, and only when you say so.
- The rules are written down. Every client gets a short AI use policy that says what AI may and may not do in the business, and who’s accountable.
- Someone keeps looking. A sample of what the AI did is reviewed every month, even when nothing has gone wrong.
What you can do this week
- List every place AI touches your business today, including staff using ChatGPT on their own.
- Run each one through the five questions.
- Mark each one automate, assist or keep human.
- Name one person who’s accountable for how AI is used in the business.
- Write down your “keep human” list and share it with the team. It’s the most useful page in any AI policy.
Sources
- Air Canada held responsible for chatbot’s hallucinations, AI Business, February 2024
- Deloitte refunds Aussie gov after AI fabrications slip into $440K welfare report, The Register, October 2025
- “Crude, cruel and unlawful”: Robodebt Royal Commission findings, Law Society Journal, August 2023
- Online product and service reviews, ACCC
- New resources on transparency for use of AI and automated decision-making, OAIC, September 2026
- Guidance for AI Adoption: Foundations, National Artificial Intelligence Centre