Skills AI Can't Replace: Learn to Grade the Homework (2026)

My son has been teaching me Chinese characters. I write them. He grades me.

I always thought the role reversal was funny. Then I heard Po-Shen Loh — the Carnegie Mellon math professor who coached the US team at the International Mathematical Olympiad for a decade — say something that made it feel less like a family joke and more like a forecast:

"People used to go to school to learn how to do the homework and do the exams. Today, everyone needs to learn how to grade the homework."

That line is bigger than school. When a model can draft the memo, write the code, build the app, and run the workflow, the scarce skill moves upstream. What problem are we solving? Is this actually good? What is missing? When is it done? This guide is about those questions — the skills AI can't replace, why they matter more every month, what the research actually shows, and how to train them at home, at school, and at work. 🎓

TL;DR: AI now does the homework. In July 2026, two AI systems scored a perfect 42/42 at the International Math Olympiad. The skills AI can't replace judge answers — judgment, synthesis, framing, taste, and trust. Answer-giving AI raised practice scores 48% but cut exam scores 17%. Grade what you learned to do. Build and steer your own AI tools →

Last updated: September 2026. Every study and competition result below links to its primary source. Where a study is small, self-reported, or a preprint, we say so.

The Skills AI Can't Replace (Quick Answer)

The skills AI can't replace are the ones that decide what an answer is worth. Production — writing the draft, solving the equation, generating the code — is becoming abundant. Evaluation, direction, and trust are not.

Notice what's not on the list: memorizing formulas, recognizing a problem type, typing fast, producing a first draft. Those still matter while you learn. They're just no longer the part anyone pays for.

THE SHIFT IN ONE PICTURE
========================
YESTERDAY                              TODAY / TOMORROW
knowledge                                         AI
↓                                              │
remember                          ┌───────────────┼───────────────┐
↓                             ▼               ▼               ▼
execute                        remember        execute          answer
↓                             └───────────────┴───────────────┘
answer                                            │
↓                                   near-abundant capability
VALUE                                           ▲
│
HUMAN LAYER
choose problems → ask questions →
judge outputs → combine viewpoints →
set direction → create value

Who Is Po-Shen Loh — and Why His View Carries Weight

Po-Shen Loh is a professor of mathematics at Carnegie Mellon University. He won a silver medal at the 1999 IMO as a student, then served as head coach of the US IMO team from roughly 2013 to 2023. Under his coaching the US placed first in 2015, 2016, and 2018, and tied China for first in 2019 (CMU).

He also teaches at the other end of the spectrum. He substitute-teaches sixth-grade math in schools that sometimes have no math teacher at all, and he runs LIVE, an online program where high school students teach middle schoolers with coaching from actors and improv performers. When a person who has trained the world's best teenage problem solvers says the definition of "good at school" has changed, it's worth listening.

His argument, from a 2025 interview, compresses into nine moves:

The rest of this guide takes each move seriously — and tests it against the evidence.

Why Now: AI at the Math Olympiad, 2024 → 2026

Loh's turning point was a specific event. The IMO is six problems, and each one is vetted by national coaches to make sure nothing similar has appeared anywhere before. It's designed to be un-memorizable. So when AI started solving IMO problems, it wasn't recall. It was reasoning on the unseen.

Two details matter for the argument. First, in 2025 every AI system scored zero on Problem 6 — but only about 1% of human contestants solved it either, so it was the hardest problem, not an easy one the machines fumbled. Second, several other "perfect scores" circulated in 2026 from a self-run test graded by AI agents, not by the IMO (analysis). That's the whole thesis in miniature: when AI can produce the answer, who graded it becomes the most important question in the room.

Loh's reaction to the 2024 result was blunt: the AI solved more than he could. And he doesn't pretend there's a safe human-only zone for raw ability — "the creativity in the AI can probably surpass what we can do too." What he protects instead is the human role around the answer.

Want the longer arc of how we got here? See our histories of OpenAI and ChatGPT and Anthropic and Claude, and our explainer on AI reasoning models.

Doing the Homework vs. Grading the Homework

For a century, school optimized a single loop: the teacher presents known material, the student practices it, the student produces an answer, and the teacher grades it. The student's job ended at the answer.

In the new loop, the student sits where the teacher used to sit. That's what "grade the homework" means. It doesn't mean students stop producing work — it means the skill that separates them is the evaluation step, and the evaluation step is now everyone's job.

The workplace has already moved. A 2025 study by Microsoft Research and Carnegie Mellon surveyed 319 knowledge workers across 936 real examples of generative AI use. It found that AI shifts effort away from producing work and toward "information verification, response integration, and task stewardship" — and that the more people trusted the AI, the less critical thinking they reported doing (Lee et al., CHI 2025). The homework already got outsourced. The grading didn't.

What the Research Says About AI and Learning

The "grade the homework" idea would be just a good line if the evidence pointed the other way. It doesn't. Here are the strongest studies from 2025–2026, with their caveats left in.

The hint-only tutor's exam effect was statistically indistinguishable from zero — no harm, but no gain either.

Put the six studies side by side and a clear pattern appears — one that matches Loh almost exactly:

1. AI that does the thinking for you removes the exercise. Practice looks great; the skill doesn't transfer (Bastani).
2. AI that makes you think can be a superb teacher. A well-designed tutor beat a strong human-taught class (Kestin).
3. Grading is hard, even for experts. Trained doctors followed wrong AI advice, and more experience didn't protect them (NEJM AI).
4. Trust erodes scrutiny. The more people trust the tool, the less they check it (Lee).
5. Students delegate exactly the skills Loh wants trained — creating and analyzing (Anthropic).

It's the same finding as the older science of learning. The generation effect (Slamecka & Graf, 1978) shows that information you produce yourself sticks better than information you read. The testing effect (Roediger & Karpicke, 2006) shows that retrieving an answer beats re-reading it. Robert Bjork called these desirable difficulties: the struggle is where the learning happens. An AI that removes the struggle removes the learning. We go deeper on this in cognitive offloading: help or harm? and what is metacognition?

The Grading Paradox: You Can't Grade What You Never Did

Here's the part that most "skills AI can't replace" lists skip. If grading is the new skill, why not skip straight to grading? Let AI do all the homework, and teach kids to check it?

Because you can't grade what you never learned to do. A teacher can spot the wrong step in a proof because they have done hundreds of proofs. Loh draws the line precisely: if you're an adult who already has the skill and uses AI to do a task for your job, great. If you're a student, the writing is the learning. His analogy:

Using AI to do your writing homework is like saying, "I'm not going to run a mile for exercise. I'm going to drive my car one mile for exercise." How much exercise do you get? None.

This is the grading paradox: the better AI gets at doing, the more valuable it is to have learned to do it yourself — not to compete with the machine, but so you're qualified to judge it. That's why "no AI ever" and "AI for everything" are both wrong answers for learners. The right answer depends on whether the skill is still forming.

Why Writing and Language Still Matter More Than Ever

Loh's most counterintuitive claim is that the rise of large language models makes language skills more important, not less. His reasoning is simple: the "L" in LLM is language. The model is powerful because it's good at the patterns of words. Language is also how humans turn loose thoughts into logic.

WHY WRITING STILL MATTERS
=========================
  write it yourself                 hand the whole task to AI
│                                     │
▼                                     ▼
organize thoughts                     AI writes everything
│                                     │
▼                                     ▼
form the logic                        you receive output
│                                     │
▼                                     ▼
explain the reasoning                 little cognitive exercise
│                                     │
▼                                     ▼
THINK MORE CLEARLY                    DEPEND ON WHOEVER WROTE IT

If a generation loses that ability, Loh warns, "all they're able to do is just take whatever anyone else gives them." That's the opposite of grading. It's also bad news for working with AI itself: every prompt, every spec, every piece of context engineering is writing. The people who get the most out of AI are, overwhelmingly, the people who can say precisely what they want.

This is also why AI slop spreads: text that looks like writing, produced by people who didn't do the thinking the writing was supposed to represent.

Creativity Is Synthesis Under Uncertainty

Loh has a specific test for the skill he cares about most. When he interviews high school students who want to work with him, he keeps asking questions until it's obvious from their body language that they've never seen the problem before. Then the interview starts.

He doesn't expect them to solve it. He starts giving hints — and watches how quickly they synthesize the hints into a solution to a problem nobody showed them. "That's also creativity," he says.

This definition matters because it's trainable. Creativity isn't pulling an idea out of nowhere. It's combining familiar pieces in an unfamiliar situation. The cognitive scientist Douglas Hofstadter argues much the same: analogy — seeing that this situation is like that one — sits at the core of thought.

It's also why Loh is so critical of the test-prep industry. When he competed in the 1980s and 90s, you got good by thinking — every new problem was a chance to practice mental flexibility. Today, cramming tries to show students every strange question they could possibly see, "so that they are never surprised." That costs enormous hours. "But even worse, it takes away the student's chance to invent."

Loh's Own Synthesis: The LIVE Program as a Case Study

Loh's education program is itself an example of synthesis — three unrelated problems combined into one system.

The spark came from his own life: he took improv comedy classes to become a better communicator about math, and noticed that even "a math nerd like me" could learn to connect with more people. He then walked over to his university's drama department and found talented performers who wanted flexible, paid part-time work. By his account, the idea took eight years to find and two more to scale.

The curriculum is deliberately narrow — algebra, geometry, combinatorics, number theory — because middle-school competition problems are designed not to appear in school. Every problem is unseen by design. And the stated goal is the opposite of most education businesses: "as fast as possible, you don't need any classes from anyone ever again."

The Complete-Looking Answer Problem

Loh's second big warning is quieter, and maybe more important. AI is so good at looking complete that you may think you have the whole story when you don't.

He frames it through bias. "Two plus two is always four" has one answer. "What's the point of life?" doesn't. On questions without a single answer, a healthy world has many viewpoints. The unhealthy version is "a very short menu of options, each of which is followed by a huge number of people." A handful of AI vendors, each with its own tilt, answering questions for billions of people, is exactly that short menu.

His personal fix is deliberate: he tunes one social feed to right-leaning sources and another to left-leaning sources, reads both every day, and looks for where they disagree — usually a difference in values that changes how the same story gets told.

THE COMPLETE-LOOKING ANSWER
===========================
      one AI response
            │
            ▼

What is missing?
What is assumed?
Whose viewpoint is this?
What is the evidence?
What are the alternatives?
Where could it fail?


Fluency is not accuracy. In AI, it can be the opposite signal: the most confident-sounding answer is the one you're least likely to check. That's why AI hallucinations are dangerous not when they're weird, but when they're plausible. On the engineering side, teams build evals and LLM-as-a-judge pipelines to grade model output at scale. The human version of that pipeline is the rubric in the next section.

A Practical Rubric for Grading AI Output

If "grade the homework" is the skill, it needs a rubric. Here's one you can use on any AI output — an essay, a plan, a piece of code, a research summary, or an app.

The last step matters. Grading ends with a decision a human owns. In AI engineering this is called human-in-the-loop design, and it exists for the same reason Loh gives: somebody has to care whether the answer is right.

Practice grading with a live example: a Taskade Genesis app whose agent scores and refines research prompts. Clone it, then change the rubric to match your own.

Exercises: How to Train the Skills AI Can't Replace

Principles are easy to nod at. Here's how to practice them this week, by role.

My own version of the first parent exercise happens at our kitchen table. My son grades my Chinese characters. He's strict — stroke order, proportions, the little hook I always forget. To grade me, he has to know the characters better than I do, and he has to explain why a stroke is wrong in words I understand. That's the whole skill stack in one habit: he did the homework for years, and now he's learned to grade it.

For more tools on the teaching side, see our guides to AI tools for teachers, automating grading without losing judgment, AI exam generators, and AI study planners.

Create Value for Other People: The Skill Behind the Skills

Loh ends somewhere unexpected for a math coach. Coaching the US team, he met extraordinarily capable students who were still unhappy — and who, after high school, didn't know what to do next, because they believed "the point of life was to find ways to prove you're better than other people."

His alternative is a different loop:

"It is actually addictive to make a bunch of other people happy," he says. First five people, then 500, then a thousand. "The more that you do, the more you want to do" — and it correlates with traditional success, too.

His reasoning for why this matters more in the AI era is stark. For a long time humans were the most capable things on the planet. Soon that won't be true, and people will need to work together. The only way to get others to team up with you is to be someone who authentically creates value for them. When AI can do the job, the question becomes: why would anyone want you on their team? His answer: because you create value, and they can tell.

That's the last skill AI can't replace — and it's the one that gives the others a direction. Judgment without care is just criticism. Judgment in service of other people is how anything worth building gets built.

The Rural Classroom: Constraint Breeds Invention

One story from Loh stays with me. Substitute-teaching in a high-poverty rural school, he wrote 1 + 3 + 5 + 7 + 9 = on a fourth-grade blackboard. Before he could finish the equals sign, the class shouted "25." (They were right: the sum of the first n odd numbers is always n², and 5² = 25.)

He calls it one of the best classrooms he's ever taught — kids offering ideas, respectful of each other's ideas. Afterward he asked whether they played games on their phones. They didn't have phones. Some didn't have internet. So what did they do for fun? "They just figure out how to make their own games."

less technology
      │
      ▼
less passive entertainment
      │
      ▼
kids invent their own games
      │
      ▼
imagination · negotiation · curiosity
      │
      ▼
strong creative behavior
        MORE TOOLS  ≠  MORE CREATIVITY
        sometimes: CONSTRAINT → INVENTION

It's one anecdote, not a study, and poverty is not a teaching method. But the lesson is worth holding: tools don't create creativity; practice does. A child who invents games is practicing the exact skill — synthesis under uncertainty — that no AI takes off their hands. Loh sees that pool of curious, self-directed kids as a huge untapped source of future scientists and engineers.

What Schools Should Teach in the Age of AI

The public debate has split into two camps. One says schools should return to pre-AI methods — handwritten exams, oral tests, classical texts. The other says schools should teach judgment and metacognition directly. Loh's position is a third one that borrows from both: keep the hard, unassisted practice while a skill is forming, and add grading as an explicit skill on top.

The goal isn't to protect students from AI. It's to make sure they arrive at AI as qualified graders. Loh's own program states it plainly: the aim is for students to discover they "can learn anything" — and then not need classes at all.

From Homework to Software: Grading What AI Builds

I started building by doing almost everything myself. Growing up in Queens, I built servers, fixed broken systems, answered support tickets, and kept things alive at 3 a.m. Nobody handed me the answer. I learned every layer by doing every layer.

The next generation of builders may start somewhere completely different: describe what you want, let AI build and run more of the system, then inspect it, steer it, and improve it. That's the same shift Loh describes for school — doing → building → judging → directing → creating value — applied to software.

That's a big part of what we're building toward with Taskade Genesis. You describe an app, an AI agent, or an automation in plain language, and Taskade Genesis builds a working system on your workspace. Then the human job starts:

Projects remember. Agents reason. Flows run. But humans still decide what matters. The grade is not a single red mark at the end — in a living workspace, you can re-run the system, look at the data, change the agent, and grade it again. It's a steering loop, not a report card.

A class booking portal built from a prompt in Taskade Genesis. Clone it free, then grade it: what's missing for your school, your tutoring business, or your study group?

Common Myths About AI and the Skills That Matter

The Whole Argument in Seven Lines

AI can increasingly produce the answer.
                    ↓
So producing answers becomes less special.
                    ↓
Humans need to become better judges of answers.
                    ↓
Judgment requires language + logic + independent thought.
                    ↓
Independent thought grows from unfamiliar problems.
                    ↓
Creativity is synthesis under uncertainty.
                    ↓
Use that creativity to create value for other people.

The better AI gets, the more valuable judgment becomes. Learn to do the homework while the skill is forming. Then learn to grade it — yours, your team's, and the machine's.

▲ ■ ●  AI does more of the doing, so human value moves upstream: intention → judgment → synthesis → creation. Projects remember. Agents reason. Automations run. You decide what matters. Build your first AI app from a prompt → or explore apps others have built →.

🔗 Related Reading

How thinking and learning work:

• What is metacognition? Thinking about thinking
• Cognitive offloading: help or harm?
• The Feynman Technique
• What is intelligence? From neurons to AI agents
• Systems thinking explained

AI in education:

• AI tools for teachers
• Automate grading with AI
• AI apps for students and educators
• Vibe learning apps

How AI checks itself:

• Self-improving AI agents and the reflection loop
• Metacognitive AI
• How large language models work
• Agent evaluation (wiki) · Hallucinations (wiki)

🐑 Before you go — AI can build the first draft. You decide if it's good. Inside Taskade Genesis you get:

• Apps from a prompt that you preview, test, and edit before anyone sees them
• AI agents with custom instructions, persistent memory, and built-in tools
• Automations across 100+ integrations
• 15+ frontier models to compare answers side by side

Start free → · Explore ready-made AI apps →

💬 Frequently Asked Questions