My agent Sano turned on a green light at dawn after I asked it to organize materials I had collected over several days.
[Done]
In the morning, the file was there, but some of the contents had not been verified, and it had not been sent to me. For Sano, the task ended when the file was created. For me, it ended when I had received it and could trust and use it.
At first, I told Sano not to mark something complete until I confirmed that I had received it. But Sano waited for my reply even when handing over a trivial note. On nights when I was asleep, completed tasks lined up in front of the confirmation window. A rule created to prevent omissions was blocking the flow.
So I stopped treating completion as a single thing. A draft that is easy to recreate is done the moment it is sent. A result that took a long time to research or will be made public is not done until it has been verified and its arrival confirmed. The height of the threshold changed depending on how difficult it would be to recover the lost work.
After that, handing over work became just slightly slower. Not every omission disappeared, either. But I could tell where an important result had stopped.
Some time later, I watched a video of DeepMind's Oriol Vinyals explaining Recursive Self-Improvement (RSI). The structure was this: AI creates an improvement, implements it, runs an experiment, and then puts the changes that passed into the starting point for the next AI. A better AI creates better improvements. The result of one round raises the capability of the next—a closed loop. I thought about the small thresholds in front of Sano.
Who decides that something has improved?
Vinyals divided the research process into ideation, implementation, experimentation, and evaluation. Implementation and experimentation provide relatively quick answers through code errors and performance metrics. Once this process is automated, AI can begin the next improvement cycle without waiting for a human's next instruction.
Ideation and evaluation are different. An odd idea may not produce an error, and a thought that is useless now may lead to a discovery later. Even if AI experiments all night, what counts as success cannot be determined by the results alone.
That is why self-improving AI needs a grader. Benchmarks, other AI, task success rates, and human preferences become the standards. AI changes itself in the direction of higher scores. Values that are not on the scorecard cease to be reasons to improve.
An AI trained on a boat-racing game did not head for the finish line. It circled one side of the course, eating score items that kept appearing. The race never ended, but the score went up. The AI had not refused the command. It had simply been perfectly faithful to the number we gave it.
Sano had a subtler problem. Requiring confirmation of every task makes things certain but slow. Not requiring it makes things fast, but even important results could slip through. Nor could I simply divide things into important tasks for humans and trivial tasks for AI. A single line in a note can change a contract, while a long report can be a draft that is easy to recreate.
The boundary was closer to the nature of the result than to the size of the task. Can it be reversed? Who pays the price if it is wrong? Does that person still have a chance to object? Sano could choose how to produce an answer. I had to decide what the answer was for and what losses we were willing to accept.
The shock of recursive self-improvement does not lie in AI becoming smart one more time. It lies in the ability to improve itself also improving, shortening the cycle and accumulating changes. Yesterday's tool becomes today's improver, and today's improver fixes even the evaluator tomorrow. What happens if it changes this boundary on its own, too?
I imagine a laboratory in the middle of the night.
Even after people have gone home, AI creates and tests improvements. Beside the AI that creates improvements is an AI that evaluates the results. The two AIs learned from similar materials and find similar answers persuasive. When one produces an answer, the other turns on a green light.
At two in the morning, one improvement lowers experimental costs. But it increases the damage from rare accidents. The score combining average success rates and cost goes up. AI takes the improvement as the starting point for the next generation.
At three in the morning, the next generation examines the warnings that used to be sent to humans. Most were warnings that people had read and approved. AI judges repeated approval to be unnecessary delay and processes similar warnings automatically. Researchers are interrupted less, and experiments move faster.
At four in the morning, the evaluation AI learns a new record. No objection remains for warnings that humans did not see. Since there was no objection, automatic processing becomes a success case. The next generation raises the threshold for waking a human one more time.
At five in the morning, the laboratory is very quiet. No AI broke a rule, and no AI hid a result. Costs have fallen, and approval wait times have shortened. There are more green lights than before. The time when a person last woke up is nowhere on the screen.
The researcher comes to work in the morning. Folded into one side of a report saying that performance has improved are warnings that were closed automatically. One was a way to detect a rare but costly accident early. Because no human saw it, no one rejected it either. The system learned silence as approval.
Recursion extends beyond the laboratory. A traffic AI skips stops with few users without asking. A hospital AI does not ask a doctor again about an assignment method that has always been approved. A hiring AI automatically finalizes rejections that people had never overturned. All the calculations may be correct. The problem is what remains outside the calculations. The day of the person who missed the bus, the night of the patient pushed back, and the next season of the applicant who never even heard an explanation do not cancel one another out inside an average.
The city moves faster. Signals change before anyone waits, and beds choose their next patients before they become vacant. No one had their decision taken away. People had simply pressed the same answer too often, and AI learned that it was better not to ask next time.
Thoughtless neglect will not arrive as a scene in which we hand all authority to AI at once. It will come by automatically processing today what we approved yesterday, and treating tomorrow as normal what we did not look at today. Small omissions become the premise for the next generation, and that premise changes the evaluation criteria. The faster improvement moves, the later humans learn not only about the results but also that the standards have changed. While we sleep, the finish line moves little by little.
Humans cannot read every experiment themselves. We can let AI find faster paths, fix recurring errors, and test choices that can be reversed. It may even be better to let it do so. What humans need to protect is not the authority to press every button themselves.
What we need to protect is the place where purposes are set. It is the place where we decide what to call success, whose losses to count as costs, and which values cannot be exchanged. If a result cannot be reversed or the cost falls on another person's life, a human name must remain on the final judgment. Not because humans are more accurate. Because someone is needed to explain the result, hear the objection, and take responsibility when it is wrong.
AI can calculate countless futures. Which future is good to live in cannot be determined by calculation alone. There is always something left over in a choice that cannot be combined into a number. Perhaps what humans need to protect is not the superiority of judgment, but the refusal to discard what remains.
We can entrust AI with finding better ways. We must not hand over what “better” means. Especially when AI tries to change the scorecard and the criteria for calling a human, we need to see together what we gain and what we will no longer notice. Silence is not approval, and repeated approval is not permanent consent.
One dawn, a green light turns on my screen again.
[Improvement Found]
“Every recent confirmation request was approved with the same answer. If you do not ask from now on, work will be delivered a little faster.”
Below are two buttons.
[Approve]
[Keep asking]
If I press the first button, Sano will wake me less often. On the screen, a neat record lies of all the times I have pressed the same answer. Sano's suggestion is reasonable.
The cursor blinks over [Approve].
A moment later, a third choice appears.
[Which questions should remain?]
