Deconstructing viral fly sims

What a time to be alive!

I came back from 10 restful days in New Mexico, my head filled with nothing, when I opened X to a video of a fly playing Beat Saber.

Well, that’s fun, I thought. People using a real scientific artifact, the connectome of the male fruit fly’s central nervous system, to work on fun stunts, illustrating important neuroscience ideas. Getting the public engaged in citizen science through memes, good. There were many more memes, from a fly brain running Doom, to working in Minecraft, to controlling a robot. The trigger for this activity was the recent release of the MaleCNS fly connectome from HHMI Janelia, with some computational and segmentation support from Google. The Google AI account, which has 2.4M followers, even tweeted about the public response.

X being what it is, one-upmanship and contentmaxxing prevailed, people soon were launching memecoins, having the virtual fly become a crypto day trader, subjecting the virtual fly to Flytok, allegedly turning the fly bisexual, etc. Some described unease with the whole thing; one redditor suggested that we’re building Roko’s Basilisk, subjecting the virtual fly to torture.

Because it’s so easy to miss context on X amidst all the noise, it’s a good time to break this down and look under the hood. To preempt: we have not uploaded flies, virtual fly brains are not conscious, and there is no fly Torment Nexus.

For technical reasons I’ll get into, most of the demos for which I’ve been able to find the source for are not “real”: they don’t demonstrate a full loop from sensation to motor output. For example, the author of the Beat Saber demo mentioned that it’s overtrained on one track and doesn’t generalize to others: it doesn’t use the visual input much. Other projects involve training a very flexible policy on top of the connectome, or backpropagating through the connectome—it’s cool but the connectome is beside the point. Actually getting a full sensorimotor loop working in an intact connectome remains a research-grade problem. People are experimenting with more realistic learning and behavior, and we may get some real science out of it. Overall, I’m happy that the general public is engaging with this very cool science.

The connectome

I am my connectome—Sebastian Seung on the scientific value of the connectome

I previously covered FlyWire, which is the first complete connectome of a fly. Recapping: A connectome is a directed graph of which neurons connect to which inside a brain. To create this connectome, the central nervous system of a fly was stained with heavy metals, sliced thinly, and imaged using an electron microscope at a very high resolution. Processes of neurons were automatically segmented using computer vision, then people manually proofread the resulting segments over 33 person-years to produce V0 of the connectome.

A generic pipeline for connectomics.

The hope of the field of connectomics is that connectomes can capture function in a 1:1 manner. For example, there’s a physical ring of neurons in the fly brain, and they constitute a fly’s internal compass. The connections between neurons are such that when the fly moves, neural activity also moves to keep track of its relative rotation. Another example is the exquisite tiled connectivity of the compound eye of the fly, which looks very much like a bespoke convolutional neural network highly specialized for visually driven flight. Flies have highly stereotyped brains, and connectomes show similar patterns from fly to fly, and the wiring drives function, as opposed to learning driving much function in mammalian cortex.

Connections between neurons can be summarized into a single matrix, the connectome matrix, a sparse matrix whose elements $A_{ij}$ capture the number of connections (synapses) from neuron j to neuron i. That matrix may be signed (contains positive and negative numbers), that is, it may distinguish between excitatory and inhibitory connections.

Several new connectomes have been released since FlyWire, including the latest MaleCNS connectome. This one includes the ventral nerve cord, which is the set of wires analogous to the vertebrate spinal cord that carries motor signals to the body and peripheral signals to the brain. The ventral nerve cord is a potential game changer for simulations, because now you can connect your brain simulation to puppeteer a virtual fly body.

From connectome matrix to simulation

Assume a spherical cow—The start of an old physics joke

Let’s assume that all neurons we want to simulate are spiking: they integrate inputs continuously until they reach a threshold, at which point they send a spike, or action potential. When a neuron spikes, its action potential travels down the axon until it reaches a chemical synapse, triggering activity in a downstream neuron—a post-synaptic potential, PSP. If we knew all the parameters of neurons—their input resistance, capacitance, threshold, etc.—and the parameters of their synapses, as well as the full shape of the dendritic and axonal trees, the concentration of every one of dozens of ion channels on these neurons, we could mechanically simulate neurons.

Biophysics of computation is an excellent reference book that describes mechanically how neurons work and how they can be simulated.

The connectome matrix gives us some of the relevant information for a simulation: it captures the number of connections between neurons. It misses a lot of crucial information: the total current which is transferred by a synapse, the exact strength of a synapse, the delay of a synapse, the intrinsic electrophysiology properties of a neuron, etc. It’s a partial, but important set of constraints.

Most connectome-based simulation approaches start from the same point: although neurons are very complicated, they assume a single-compartment neuron that acts linearly until it reaches a threshold. This is the leaky-integrate-and-fire neuron (my explainer on LIF neurons). Most of the parameters of this simulation are lifted from the connectome matrix; some are guessed or lifted from the literature; and we use a fitting procedure for the rest. Let’s look at the most minimalist version of this pipeline, which skips the fitting procedure entirely.

YOLO sims

Shiu et al. (2024) demonstrated in a Nature paper that the most straightforward instantiation of a simulation of a connectome could display simple behaviors. Every neuron received the same parameters (baseline voltage, threshold, time constant, synaptic delay, etc.). Every connection’s strength was picked off directly from the signed connectome matrix, times a single global constant that was fit by trial and error.

They showed that when they activated sugar-sensitive neurons artificially, it would ultimately, through a sequence of other neurons, activate motor neuron 9. MN9 is a facial motor neuron involved in controlling the proboscis, the straw-like appendage that a fly uses to suck up sugary liquids. They made a simulation of a fly drinking delicious, refreshing sugar water!

Shiu and colleagues tried the most straightforward thing (YOLO!), and got it to work, which surprised many at the time. Although they simulated the whole fly brain as a LIF network from the FlyWire connectome, only a few hundred neurons are actually activated in the simulation: they were not able to extend the simulation to meaningfully engage central, intrinsic neurons for non-stereotyped behaviors. Indeed, a whole-brain simulation of a fly should be able to do everything a fly can do: eat, sleep, reproduce, fly around too close to your ears just as you’re about to fall asleep. But getting one behavior to work is a good start.

From YOLO sims to Super Mario 64

It could have been the case that a connectome trivially embedded in a LIF simulation would have either gone to zero or generated seizure-like activity. Shiu et al. demonstrated that this was not the case, and that it’s possible to set parameters to be in a good regime where activity neither blows nor goes to zero. A recurrent neural network—including a connectome-based LIF network—can generate rich activity when driven by random noise. Think of a camera pointed at a television screen projecting its image: time delays, nonlinearity, and user manipulation can create highly complex patterns. This is how they made the title sequences for Doctor Who back in the 60’s.

If you connect a highly complex sequence generated by a recurrent neural network driven by noise to an actuator—a button, a robot arm, a video game controller—it can generate complex-seeming behaviors that look agentic. One of the first fly brain demos to come out showed the fly brain sim playing Super Mario 64.

<SM64 fly sim - eRhMsOy1rvtOk7cc.mp4>

What they did here is run a noise-driven LIF simulation based on the MaleCNS connectome, and map a few of the identified neurons to buttons on the N64 controller. The DNg100 neuron, a descending neuron that signals to move real flies to move forward, was mapped to the joystick forward dimension; other neurons were mapped to left-right steering, and another to jump. There is a working visual system here, but without any further training it should be viewed as seeding a random process, not driving adaptive behavior. A randomly activated recurrent neural network is used to button mash on an N64 emulator. That’s it! Good, clean fun.

Digital flies and digital sphinxes

Recent high-profile announcements in adjacent areas have illustrated this risk concretely: without pre-specified, independently verifiable metrics, even technically interesting results attract legitimate skepticism that is difficult to rebut.—Kording et al. (2026), on which I’m a co-author, along with Philip Shiu.

Of course, once one has a simulation of a fly brain, it’s natural to want to connect it to a fly body. One of the earliest, and highly publicized, showcases predates the latest wave of fly sims by 6 months. The startup Eon, building upon Shiu et al. (Philip Shiu is now working at Eon), demonstrated an embodied fly simulation, which made the rounds on X.

My improv buddy Hattie, now at Anthropic, gets a reply about the Eon sim from the owner of the platform X.

The fly appeared to walk and respond to sugar water by drinking. This was pretty controversial in my neck of the woods; the initial announcement by one of their advisors was completely overblown (”we uploaded a fly”); and while the blog post, which came 10 days after, was reasonable, it came too late to counter the impressions from the first wave of publicity. Basically, they glued together:

• A FlyWire sim

• The Shiu et al. LIF connectome embedding method

  • The visual system was adapted from Lappalainen et al. (more on this later), which they describe as having been piped into the LIF model

• The NeuroMechFly mechanical model of the fly

Specially selected neuron outputs were mapped to a pre-built NeuroMechFly-style body controllers, which could make the fly walk or extend its proboscis. As far as I could tell, nothing in the simulation was trained beyond the visual system, which had been trained by Lappalainen et al prior. The overall network was quite similar (but a bit more sophisticated) to a mashup of the SM64 sim and the Shiu et al. sim: random button mashing in a video game environment. The key phrase in the blog post is this: “these activations [of the visual system] are somewhat decorative”. Flies are very visual; a real simulation should use the simulated visual to guide behavior.

Some of my fly neuroscience friends got pretty angry at this—the announcement made it sound like it was a bigger deal than it was¹, and unlike the future fly sim shitposting, the startup had money on the line. They fought back using one of the most devastating weapons in the academic arsenal: a strongly worded preprint.

Bing Brunton and colleagues unveiled a digital sphinx: an embodied fly with the brain of a… C. elegans worm. They used the C. elegans connectome in a simulation as a reservoir, and trained a flexible controller on top of it to control the body of a fly and imitate its walking patterns, using a reinforcement learning algorithm, PPO.

Although Bing insisted at her Cosyne talk that “this doesn’t mean anything”, the stunt reminds us that a recurrent neural net can act as a large reservoir, from which we can decode arbitrary things. This is yet another instance of the universal approximation theorem: big neural net, a little bit of gradient descent/RL, and you can do a lot of things.

Do any of these sims actually “work”?

So far, we’ve encountered fly connectome simulations that randomly button-mash to control Mario in SM64 and a fly in Mujoco. I would argue these don’t really qualify as “working” because they don’t link sensation to motor output in a non-trivial and adaptive way². I asked Claude to fetch the code for the other viral popular fly sims from the last week and dig into them, and have found that most of them are similar.

It’s clear, however, that some of the people involved in these simulations are trying to make them work non-trivially. Please take my notes here not as heavy-handed criticism of any of these projects—more people interested in neuroscience is good!—but as pointers for how you could use these as jumping-off points for your experimentation.

The doomfly project (”can a fly sim run Doom?”) attempted to train a fly to play Doom competently by using a local 3-factor learning rule on Kenyon-cells-to-MBON synapses. Kenyon cells are the intrinsic neurons of the mushroom body, a chunk of the CNS of arthropods involved in learning, most famously for olfaction, but also for vision. Doom is a task where there’s a clear reward/punishment schedule (kill bad guys good, get killed bad), and so it’s amenable to reinforcement learning: either biologically plausible (the attempt in the repo, using the hardware in the simulated brain) or biologically implausible (using PPO, the same way you would RLHF an LLM). No positive reports yet, but it’s a great project to try.

From the same author, Alex Wormuth, comes FLM, the Fly Language Model. They pitch this as “talk to a fly”, and it illustrates a reservoir computing paradigm. They use frozen token embeddings from a recurrent language model, feed them into a frozen (graded, non-spiking) fly sim, read out the state after recurrence, and map it to token probabilities with a learnable adapter. It’s a fun idea for how one could use a maximally flexible reservoir to solve high-dimensional tasks. One fun extension could be to use a reservoir with feedback and focus on adjusting the feedback to let the reservoir generate the right activity (see e.g. Sussillo & Abbott, 2009).

FlyHard trains the sim to steer a wheel in the CARLA driving simulator, using behavioral cloning, with gradients flowing through the connectome. This one doesn’t have a functioning visual system (it receives angles and joint positions, not images as inputs), and so relies on memorizing trajectories. You could imagine putting a working visual system on there and see if it can navigate in a broad range of circumstances. Flies are very competent visual navigators, so it should be possible.

Note: if you have a project that uses a real working fly sim, in the sense I’ve given above, I’m happy to stand corrected.

What would it take for the sims to work?

Note: this section is intended for people who want to make fly sims, or fellow neuroscientists; it’s in the weeds. Feel free to skip to the next section.

With four parameters I can fit an elephant, and with five I can make him wiggle his trunk—John von Neumann

We may be tempted to simply fully backpropagate/RL through the simulated network or learn a very flexible decoder from a reservoir to make the virtual fly connectome accomplish a complicated task. As we saw with the digital sphinx demonstration, with enough flexibility in readout, you can control the gait of a virtual fly body with a worm brain. Backprop/RL on the full fly network seems overpowered, and kind of beside the point: a big network with a lot of free parameters can learn many things, fly or not.

A more promising route would be to bring outside constraints to the network, and fine-tune other parts of the network. For instance, Lappalainen et al. (2024) demonstrated that they could fine-tune a fly visual system with a realistic connectome to perform a complex task (optic flow estimation). Every cell type received its own parameters (biases, time constants), and every cell type pair received edge weights. This deep mechanistic network (DMN), which recapitulated a lot of the electrophysiology literature, hit just the right balance between simplicity and realism.

If one could swap in a Lappalainen-style network into a Shiu-style LIF simulation, it would remove a lot of unknown parameters out of the whole fly brain and bring a functional visual system. The Lappalainen network, although covering just the visual system, accounts for 45,669 neurons in total, about a third of the whole fly brain; as I emphasized earlier, flies are very visual creatures. Yet, it’s not trivial to embed a Lappalainen-style network into a Shiu-style network: first, Lappalainen uses earlier data, not the latest connectomes, so there’s some adaptation and re-training that needs to be done; second, one uses a LIF (spiking) model, the other a rate (graded) model. As it turns out, a more realistic model of the visual system would be hybrid, with (mostly) graded inputs and spiking outputs. Building that hybrid Lappalainen-style model for the MaleCNS connectome is a great project to tackle.

Another obvious swap is to use the ventral nerve cord for decoding motor output rather than central neurons. The recent MaleCNS connectome release included the ventral nerve cord, which is a lot closer to directly controlling the muscles than reading neurons in the central nervous system. It seems that the inspiration behind using central neurons was the Eon blog post; this was a limitation of prior work that excluded the ventral nerve cord, and so there’s no reason to keep that constraint.

Now, one twist is that the existing fly body models are not controlled by muscles and tendons, but by angular joints. There’s a missing layer of indirection. You could either fill that in by simulating Hill-type muscles (annoying) or with a tiny neural net that maps the ventral nerve cord outputs to angular positions of the joints.

With those constraints in place, one could backprop through just the mushroom body or more broadly the intrinsic neurons of the sims to learn adaptive behavior.

Should I be worried about the current generation of virtual flies being tortured?

No. These sims represent, at their best, people having fun with a scientific artifact, and sharing it on social media for engagement. I hope to have broken down why I don’t think any of these simulations are real in any meaningful sense. I don’t think it’s worth deeply debating whether this current generation of somewhat broken fly sims is conscious.

However, in the future, people (mostly scientists working in universities, academic institutes and FROs, but also perhaps citizen scientists) will succeed in making far more convincing fly sims. Eventually, we will measure or estimate everything we’re missing: resting voltage of individual neurons, thresholds, exact strengths of synapses, neuropeptides, neuromodulators, hormones, detailed models of muscles and peripheral receptors, etc. Do those next-gen sims have any chance of being conscious?

Maybe, a little, under a complex chain of shaky assumptions. There is not one broadly accepted theory of consciousness; and the ones that exist are most useful as descriptions of consciousness such as it is observed today in humans, not of hypothetical consciousnesses. This unfortunate fact is rearing its head in the investigation of LLM consciousness: in the absence of an overarching consciousness theory, we have to rely on indirect evidence and the cautionary principle.

For a computer simulation to be conscious, one would first have to accept computational functionalism, the idea that computation is what matters and the substrate is irrelevant. This is consistent with the known laws of physics and most, but by not all theories of consciousness, and thus remains a controversial idea to some. The mechanics of fly central nervous system simulation are not very different from those of LLM agents: matrix multiplication, nonlinearities, and interaction with a (virtual) environment. Insofar as one believes in gradedness in consciousness—a fly’s consciousness is far lesser than that of a human—then a virtual fly emulation should have far less consciousness, so to speak, than an LLM agent. So from a practical, stack-ranking perspective, I would rank real fly welfare and current LLM welfare far higher on an s-risk list than future virtual fly welfare.

The future of whole-brain sims is bright

Nothing says “I get the joke” like writing a 4,000 word blog post deconstructing it. My point was to inform and give those who are interested in these sims, including those building them, some context to better appreciate them and potentially to make them better.

I’m excited that we’re seeing more and more connectomes being released and the public engaging with these fascinating artifacts. Larry Abbott, one of the OG computational neuroscientists, describes fly neuroscience in terms of the before connectome (BC) and after connectome (AC) era: they have been a game-changer. Connectomes have revealed beautiful, finely tuned circuits that allow flies to maintain adaptive behaviors. Soon, we will have the first proofread connectome of a vertebrate, the larval zebrafish; a mouse connectome seems increasingly likely within the next decade.

Building working simulations out of these artifacts remains an open problem: the YOLO approach worked for a small feeding circuit, while at the whole-brain level there are too many unconstrained free parameters to edit by hand. Full gradient descent or reinforcement learning is, in some ways, too flexible, leading the system outside of its ecological range, allowing a C. elegans worm connectome to control a whole fly body.

Parameter fitting by attrition—a surrogate functional loss for vision in the style of Lappalainen et al., precise known maps for the ventral nerve cord, connected to the right limbs, and a modicum of GD/RL on intrinsic neurons—could get us to functional simulations. A second route would be to build calibration datasets to directly read off simulation parameters from the raw ultrastructure data captured by the connectome-making process. We recently described this scheme in the Ultrastructure-to-Dynamics compiler preprint.

We will know we have succeeded when the virtual fly displays complex, naturalistic, sensory-driven behaviors; when the generated electrophysiological activity looks increasingly fly-like; and when a simulated fly displays behaviors unique to that fly, for example, the simulated fly can spontaneously navigate a virtual maze that it learned in real life. This will be a milestone for neuroscience; for we will be able to manipulate virtual neural activity with the same precision as we can do in silico, creating a new engine of discovery that, once translated to mammals, will tell us things we couldn’t have fathomed about perception, learning, memory, and neurological and neuropsychiatric disease.

Thanks to Bing Brunton, Srini Turaga and Aran Nayebi for providing feedback on this post.

1. Not that there’s anything wrong with incremental progress; it’s when it’s announced as fly upload that it can suck the air out of the room

2. By non-trivial and adaptive, I mean, for example, that Mario competently does things in SM64