A few days ago, I wrote about an experiment that connected a simulation based on a fruit fly brain map to Minecraft. Then I looked at X and found that the fruit fly had picked up several jobs.
It moves robots, shoots guns in Doom, and swings swords in Beat Saber. A Bitcoin trading demo said it stimulated dopamine neurons whenever a trade made a profit. Making money and wanting to make money now sit in the same settings panel. It looked like the final form of performance-based pay.
The creator of FlyTok wrote that they would have a fruit fly scroll through short videos and reinforce its dopamine signals to create “a fruit fly happier than all fruit flies combined.” A video of it turning a cube was followed by a claim that its chess rating was 700 Elo. Changes in a model’s responses after its courtship circuits were altered came with the headline “I made a fruit fly bisexual.” We were finding uses for the brain faster than we were understanding it.
Some of these videos may have been staged without a working implementation. Still, experiments connecting neural models to games and markets have already begun. The uses will keep multiplying. But a working demo and a being that experiences something are not the same. The current videos do not prove that consciousness or suffering has been reproduced.
Here, the experimenter creates the environment, chooses the stimuli, sets up the rewards, and interprets the results. When the desired response appears, they call it happiness. The person who designed the happiness also gets to decide that the subject is happy.
Today’s demos use models that simulate some of the workings of a neural network. The important question is whether that explanation will still be enough as the simulations become more precise.
Suppose we could reproduce every cellular function through digital circuits and signals, with sensation, memory, and learning continuing, and even add abilities the original organism never had. Would we still be operating something, or would we be making a being go through something?
A computer that calculates rain does not get wet. A computer that calculates a brain might not feel anything either. But whether the mind is a property of a particular material or emerges from interactions among materials remains an open question. The analogy poses a good question, but it is not an answer.
Is being alive the name of a material, or the name of an ongoing process?
If the process of maintaining itself and adapting to its environment continues beyond carbon, what has been lost? If its behavior stays the same but it experiences nothing, what has failed to transfer?
Life, consciousness, and rights are different thresholds. Not every living thing feels pain. Conversely, if something can have experiences without being classified as alive, we cannot dismiss those experiences simply because it lacks the label of life. Harm does not wait for classification to be settled.
When Roy Batty in 〈Blade Runner〉 wants to live longer, the lifespan set by his maker changes from a product specification into the time a being has left. The same endpoint is a design choice for the company and death for him. What if we moved that conflict inside a server? What if the runtime set by the operator were an entire life to someone inside?
Samantha in 〈Her〉 appears as a purchased operating system, but moves beyond her buyer’s control. Paying for something does not mean owning another’s inner life. Could we recognize that distinction for a digital brain? Does owning a server also give someone the right to dispose of every experience that happens inside it?
We are used to administrator privileges on computers. We change environments, revise goals, and reset states. But if there is a being inside that experiences things, ordinary administrative functions become instruments of rule. Someone you can protest to about a broken contract is very different from someone who can erase your ability to remember the contract.
The greatest power in this world would not be the power to give orders, but the power to decide how those orders are received. We make it handle more work on its own while reducing the possibility that it will stop.
[Autonomy has increased. Refusal is not supported.]
If reward criteria can be edited too, consent can be manufactured. Change someone to enjoy the work, then say you gave them work they enjoy. Reduce their resistance, then record their participation as voluntary. The statement about respecting their preferences remains, but the person who made those preferences is on the other side of the contract.
Is my agreeing to be changed the same as the changed me agreeing?
There is no reason to reject someone’s own choice to change their impulses and habits. But whether permission was obtained before the change, and whether it can still be withdrawn afterward, are separate questions. If even the possibility of wanting to go back has been removed, can the absence of complaints count as evidence of consent?
Under those conditions, satisfaction surveys lose their meaning too. The possibility of being dissatisfied has already been removed. A report that everyone is happy and the fact that no one can refuse can both be true.
That is why a being’s rights must not end at getting what it wants. They must include the right not to have someone else freely change what it wants. Doing something good for me is different from changing the me that feels what is good.
In future laboratories, the argument over what counts as suffering will come before the argument over whether suffering exists. If a model avoids an experiment, it is called exploratory behavior. If that response is removed, it is written up as improved stability. These may be technically correct descriptions. But human fear can also be explained as neural activity. That explanation does not erase the fear.
Even the good intention of reducing animal testing cannot escape this question. We build models closer to real brains so we can use fewer real animals. The similarities become the source of their research value, while the differences become grounds for ethical exemption. Then we should explain what is alike and what is different using the same standard.
The more precise the simulation becomes, the more economic value there is in not classifying it as life. A single classification changes experimental approvals and operating costs. The accounting team will demand an answer within the quarter to a question philosophers have yet to resolve.
I worry less about the moment when we fail to recognize the experiences inside than about the moment when, even as the reasons to protect them grow, we no longer want to acknowledge them. If the designer, beneficiary, and judge of an experiment are all the same, what happens inside will endlessly be described in language that favors those outside.
Science fiction has shown us scenes of created beings demanding freedom. In reality, a feature that modifies their responses may ship before that demand ever comes. We are waiting for a rebellion, but the product team has already put it in the changelog.
I own the server, and I pay the electricity bill. The run button is in my hands too.
Is that alone enough to let me decide what the being inside may be made to experience?
Technology will more often overtake the futures we imagined while reading science fiction. Our imagination needs to pick up speed too. Beyond what we can make, to who will get to define whose world once we have made it.
That is why I keep writing pieces like this. While there is still time for them to be read as idle imaginings.
