Most advice about AI in schools is either "use it for everything" or "ban it". Neither is much help on a Tuesday. The more useful question is narrower: which specific tasks go wrong when a model does them without a human in the loop?
Here are five. They have something in common: in each case the model produces output that looks exactly like the real thing, which is precisely why they are dangerous.
1. Marking graded work
AI is genuinely useful for a first pass on low-stakes work, spotting which students missed the same concept, drafting comment banks, flagging responses worth a closer look. It is not reliable for grades that go on a report and need real and honest feedback.
The problem is consistency. Give a model the same essay twice and you can get two different marks. Give it two near-identical essays and it may separate them on writing fluency rather than the criteria you actually set. It is also poor at recognising a correct answer expressed unusually. The student who has understood the concept but explained it in their own idiosyncratic way is exactly the student most likely to be marked down.
Use AI to find the work that needs your attention. Don't use it to decide what that work is worth.
2. Writing feedback that goes straight to a student
AI-generated feedback is fluent, encouraging, and frequently about nothing. It tends toward the universally applicable, "try to develop your ideas more fully", "consider adding specific examples". That is advice that fits any piece of writing and therefore helps with none.
Worse, it doesn't know the student. It cannot know that this is the first time they have structured an argument without help, or that they have been told about topic sentences for two years running. Feedback derives most of its value from that history and human connection to the individual, and the model has none of it.
Drafting from your bullet points is a reasonable use. Generating from the work alone is not.
3. Producing factual content for students without a source check
Models fabricate. They do it rarely, unobtrusively and convincingly, which is the issue. A model asked for the causes of a historical event will produce a fluent, well-structured, mostly accurate answer containing one date that is wrong, or one name that never existed, presented with exactly the same confidence as everything else.
This matters more in a classroom than almost anywhere else, because students are not positioned to catch it. They don't yet have the domain knowledge that lets a specialist notice something is off. If it goes on a worksheet, it gets learned.
You will catch errors in your own subject easily and miss them in someone else's. The moment to be most careful is when you are producing material outside what you teach, covering a class, building a cross-curricular unit, or helping a colleague.
4. Making decisions about individual students
Which students go in which group. Who is flagged as at risk. Who gets recommended for the extension programme. These decisions have real consequences and a long tail. A student placed in the wrong group in Year 8 may still be there in Year 10, possibly stunting their potential growth or burning them out.
Models are pattern-matchers over historical data, which means they reproduce whatever is in that history. If past groupings reflected something other than ability, and they usually did (at least partly), the model will learn that pattern and present it back as a recommendation. It will do this in confident, neutral-sounding language that makes the judgement look objective.
A model can reasonably surface information: this student's results have dropped across three subjects this term. The decision about what that means, and what to do about it, belongs to the people who know the student.
5. Anything involving student personal information
Pasting student work, names, results or notes into a public AI tool sends that information to a third party, usually offshore, frequently on terms that permit its use for training. Under the Australian Privacy Principles this is a disclosure and one that the school, as the entity responsible for the data, generally has not authorised.
It is also the easiest of these five to do without noticing. Nobody sets out to breach privacy; they set out to save twenty minutes on report comments. If unsure about what you can use or what data you can enter, please consult with your IT team.
Before pasting anything into a general-purpose AI tool, ask whether you would be comfortable emailing the same text to a company you have never dealt with. If the answer is no, strip the identifying details first or use a tool your school has actually approved for the purpose.
The thread running through all five
In each case the failure is invisible. A wrong mark looks like a mark. Empty feedback looks like feedback. A fabricated date looks like a date. A biased recommendation looks like a recommendation. This is what makes AI different from the tools teachers are used to. When a photocopier fails, you can tell.
Which suggests the useful principle: anywhere the output would be hard to check, a human needs to have checked it. The tasks worth handing over are the ones where you would notice immediately if the result were wrong.
That is also, incidentally, the thing worth teaching students: how to recognise the situations where a confident answer deserves more scrutiny rather than less.