We think that the path forward is through distillation of neural data; and the most impactful path for neuroAI is toward AI safety. In this blog post, we unpack how we came to that conclusion; how the emulation and distillation paths to AI safety differ in their payoff timelines; what we can hope to learn from the brain’s representations; and why we think AI safety is over-leveraged on a handful of approaches.

Amaranth Foundation Blog

Towards Magnanimous AGI

The design space of general intelligence is vast, and we have no idea how much of it is safe beyond the narrow region that we humans inhabit. We are racing towards Artificial General Intelligence (AGI) without building the guardrails necessary to prevent catastrophic risk. Because the human brain is the only broadly cooperative general intelligence we k…

Read more

a month ago · 11 likes · 1 comment · James Fickel, Patrick Mineault, Joanne Peng, and Amaranth Foundation

Our goal at the Amaranth Foundation is to bring about a positive transition to AGI. We’ve got many more posts in the pipeline covering everything from brain aging and longevity to abundant solar energy. Hit the subscribe button over there to get these updates.

It turns out that evolution has (at least) two different tricks to encode intrinsic circuits: wiring rules and wireless (neuropeptide) connection rules. I present how these different rules interact in a circuit in the hypothalamus that controls aggression. Let’s go!

Cell Types Determine Wiring Instructions

Dániel and Albert-László Barabási propose a formal framework for understanding how genetic identity translates into connectivity (Barabási & Barabási, 2019). In their model, each neuron possesses a “barcode”—a combinatorial pattern of transcription factors that defines its cell type. Wiring rules then operate on these barcodes through what they call biclique operators.

A biclique operator recognizes matching patterns in source and destination neurons and generates connections between all neurons matching those patterns. Consider a system where neurons are identified by three binary transcription factors. An operator might say “connect all neurons expressing TF1 and TF3 to all neurons expressing TF2 and TF3”. The key insight is that a single compact genetic rule can specify many connections simultaneously. Anything from OR gates to AND & NOR gates, combined with Boolean masks, can be expressed this way.

The Barabásis tested this model against the C. elegans connectome. The key test is that the pattern of connections in this model is highly non-random: cells of the same type should connect preferentially to the same target cell types. They found these patterns—biclique motifs—at a far higher rate than what random or models would predict. Moreover, neurons within these bicliques shared statistically significant gene expression patterns, suggesting that common genetic factors indeed drive synapse formation within these subgraphs. Similar patterns appeared in C. intestinalis and drosophila olfactory circuits.

The concrete mechanism for biclique matching is not resolved in this computational paper. However, the discussion hints that it could involve cell adhesion molecules (CAMs) and receptors. When neurons of different types come into contact during development, surface proteins recognize each other through receptor-ligand interactions. If neurons of type A express receptor X and neurons of type B express ligand Y, and the X-Y combination triggers synapse formation, then all A will end up connecting with B (provided they’re within range).

Indeed, a long line of work finds that CAMs allow different neuron types to very precisely find each other, even through the clutter of the neuropil. There’s a great paper from Yoo et al. (2023) that finds conceptually similar wiring rules in the fly’s visual system, where one has access to both transcriptomes and connectomes. Here, it’s the Side and Beat receptor-ligand pairs that determine with great precision how different motion-selective neurons connect. This is one way evolution writes circuit diagrams into the genome: by specifying the molecular handshakes that create connectivity patterns.

Connectivity Is Function

A deeper principle underlies the cell-type framework: connection topology determines computational function. By wiring up neurons in just the right way, we can create integrator, differentiators, and all sorts of bespoke circuits. A review by Khona and Fiete catalogs how different connectivity motifs produce different attractor dynamics (Khona & Fiete, 2022):

• Point attractors arise from recurrent excitation balanced by global inhibition, implementing memory systems like the Hopfield network.

• Line attractors—continuous manifolds of stable states—require precisely tuned excitation-inhibition balance with quasi-linear neural responses, enabling integrator circuits.

• Ring attractors emerge when neurons form cyclic connectivity patterns, allowing, for example, head direction cells to encode angular position continuously.

• Winner-take-all dynamics emerge from mutual excitation within groups combined with cross-group inhibition.

The implication is profound: by specifying which cell types connect to which—via CAM operators in the Barabási model—the genome encodes not just anatomy but dynamics. A circuit that integrates, oscillates, or makes winner-take-all decisions does so because its connection topology was specified to produce those dynamics.

The Wireless Connectome: Neuropeptides as Parallel Channels

The Barabási model describes the wired connectome—synaptic connections determined by cell adhesion molecules. But there is a parallel system equally important for genome-specified communication: neuropeptides, which constitute what we might call the wireless connectome. Some neuropeptides are household names, like oxytocin (often referred to in the popular media as the love hormone, although it’s much more complicated), glucagon-like peptide-1 (GLP-1, the receptor for blockbuster weight-loss drugs like Ozempic), or β-Endorphin. Others may be familiar to neurophysiologists, like SST and VIP, used as markers for subtypes of inhibitory cortical neurons.

When I first started learning about neuropeptides, it broke my mental model of how neurons work. Like neurotransmitters, neuropeptides come in pairs: ligands (encoded as neuropeptide precursors in the genome, NPPs) and receptors (G-protein coupled receptors, or GPCRs). However:

• They don’t get released at synapses. They get packaged up in dense core vesicles and are released from the cell body or from axons.

• They diffuse willy-nilly in the extracellular space through Brownian motion. Hence, “wireless transmission”. Their effective range is a few hundred micron radius ball around the diffusion location, though that can vary by neuropeptide species.

• They are not tightly recycled or degraded like neurotransmitters. They can persist over minutes.

• Sensitivity at the receptors is sky-high: in the nano-molar range.

• There’s O(100) neuropeptide systems in the brain, covering, between protopeptides and receptors, upwards of 500 genes—about 2% of the human genome.

• They’ve been around forever, predating neurons.

Whenever you think that the brain is really complicated, it usually turns out it’s more complicated than that.

A parallel layer of communication

Neuropeptide signaling operates on slower timescales and can affect neurons beyond the reach of synaptic wiring. If we think of cell adhesion molecules as determining who can wire to whom, neuropeptide ligand-receptor pairs determine who can signal to whom through the extracellular medium. The combinatorics are rich: with dozens of neuropeptide families and their receptors, neurons can express different “chords”—combinations of ligands and receptors that define their participation in overlapping signaling networks.

Strikingly, both the Allen and Broad Institute transcriptomic atlases found that neuropeptides are a major contributor to splitting transcriptomic clusters (Yao et al., 2023; Langlieb et al., 2023). What distinguishes one hypothalamic cell type from another at the molecular level is often which neuropeptides they make and which neuropeptide receptors they express. This suggests that functionally distinct neuron types are differentiated by neuropeptide “chords”, particularly in the Steering Subsystem where bespoke circuits require precise signaling specificity.

Integration-by-precise-wiring vs. integration-by-soup

You can see these very different methods of building circuits dramatically illustrated by line attractors. In dynamical systems, line attractors are characterized by a tight line of attractors—a line of marginal stability. Position along this attractor line tracks a single variable, and the attractor naturally remembers this variable over time, and so are ideal to accumulate evidence. They’ve been implicated in evidence accumulation in decision-making tasks, to track a slowly decaying variable like aggression or sexual receptivity, and in machine learning, to track complicated judgements about whether a sentence has positive or negative sentiment.

You have to work hard to make a good line attractor with wired connections. Khona & Fiete (2022) present one template:

Two neuron groups with in-group excitation and across-group inhibition, precisely tuned interaction strengths and quasi-linear neural input–output responses can counteract activity decay in the network and produce persistent activity over a continuum of activity levels in the two populations, defining ramp-like neural tuning and a line of attractor states. (emphasis mine)

Of course, it’s not impossible to learn a line attractor—RNNs trained to accumulate evidence will learn line attractors—but it requires precise tuning.

By contrast, building a line attractor is trivial to build with neuropeptides. Neuropeptidergic signaling provides a natural mechanism for analog integration—the slow accumulation of signals over time. When neurons release neuropeptides into the extracellular space, those molecules persist far longer than the millisecond timescales of synaptic transmission. A population of neurons releasing the same neuropeptide create a shared pool that integrates their collective activity; neurons expressing the receptor effectively “read out” this integrated signal. I call this process of analog integration of diffusable signals in the extracellular space integration-by-soup (I hope it catches on).

Integration-by-soup is a less flexible mechanism than integration-by-precise-wiring: diffusion constants are hard-wired; it can cause interference across space. For things that relate to survival, however, they can be just the right mechanism, provided that there is a fixed number (~100) of slowly changing state variables that need to be tracked.

The VMH Line Attractor: A Case Study

A hypothalamic circuit that tracks aggressivity illustrates how wired and wireless mechanisms could work together to create a line attractor (beautifully reviewed here). Nair et al. (2023) first described neurons in the ventromedial hypothalamus (VMH) expressing estrogen receptor 1 (Esr1). These neurons encode an aggressive internal state during both fighting and observation of fighting: dynamical systems analysis revealed an almost perfect line attractor in state space, with very slow temporal decay (>50s). In Vinograd et al., 2024, they identified two functionally distinct subpopulations within this genetically defined cell type:

• x₁ neurons form the “integration dimension”—they show slow dynamics with long time constants, accumulating activity during aggressive encounters. Activity projected onto this dimension persists after stimuli are removed, ramping up along a line attractor that correlates with escalating aggression from sniffing to mounting to attack.

• x₂ neurons form an orthogonal dimension with faster dynamics. These neurons show transient responses time-locked to stimulus onset but do not integrate over time.

The critical finding concerns functional connectivity: single-cell optogenetic stimulation combined with calcium imaging revealed that x₁ neurons are functionally interconnected with each other but not with x₂ neurons. When x₁ neurons are stimulated, other x₁ neurons show increased activity (functional connectivity within the ensemble), but x₂ neurons do not respond. Conversely, stimulating x₂ neurons produces only transient off-manifold perturbations, and the neural trajectory returns to the line attractor rather than integrating.

Important technical distinction: functional connectivity between x₁ neurons means that when some x₁ neurons are zapped, others activate; it does not say anything about whether they’re directly connected, indirectly connected, or wirelessly connected. You would need electrophysiology and/or connectomics to infer anatomical connectivity.

Two ways to connect: choose one

The circuit could either be mediated by some bespoke wired circuitry or some bespoke wireless circuitry. Let’s look at them one by one. In the Barabási model, this circuit might arise from biclique operators:

1. All VMHvl Esr1+ neurons share a common transcriptomic identity that positions them in the same nucleus and allows them to form synapses with each other.

2. Within this population, additional transcription factors or cell surface molecules distinguish x₁ from x₂ subpopulations. If x₁ neurons express a specific adhesion molecule or receptor that x₂ neurons lack, a biclique operator can generate exactly the dense within-x₁ connectivity observed.

But there’s another way of doing this, this time wirelessly: if all the x₁ cells express some neuropeptide receptor, they could accumulate aggression information from upstream sensory neurons via integration-by-soup. So which one is it?

It seems in this case that it’s wireless integration-by-soup. In companion work, Mountoufaris and colleagues showed that knocking out neuropeptide signaling (OXT and AVP receptors) in VMHvl neurons eliminates the line attractor dynamics (Mountoufaris et al., 2024). However, the authors note that “that does not exclude a contribution from recurrent glutamatergic excitation in the ventromedial hypothalamus, as in line attractors that mediate cognitive functions on shorter time scales”. In other words, it could actually be a little bit of both, but it definitely looks like neuropeptides are involved.

I don’t want you to come off with the impression that neuropeptides are the source of all line attractors in the brain; far from it! Line attractors are ubiquitous, and those that are involved in e.g. decision-making and evidence accumulation in the cortex are likely done through conventional wiring. However, especially in parts of the brain that are part of the steering subsystem, neuropeptides can do computational heavy lifting beyond their reputation as neuromodulators.

This example illustrates the deep connection between cell types, connectivity, and computation. The genome doesn’t specify “build a line attractor for aggression.” It specifies molecular identities (Esr1+, plus whatever distinguishes x₁ from x₂) that, through wiring rules and neuropeptide channels, generate the connectivity topology that implements line attractor dynamics.

More epicycles

I presented a binary story of wired vs. wireless connectomes. Of course, the real story is far more complex, with the gnarliest complication being space. Distance strongly constrains which neurons connect with which. Layout is ultimately encoded in the genome; and the genome dictates axon guidance, through different guidance molecules like netrins, slits, semaphorins, and ephrins. That means transcriptomic identity also influences layout and guidance signals. it could thus be said that cell types influence connectivity through (at least!) three interacting means: local wiring, wireless neuropeptides, and global layout.

Not only that, but these factors are dynamic! Neurons migrate during development, which means that local connections can translate into long-range connections. Transcriptomic signatures also change during development. It’s possible, for example, that CAMs are expressed only at an intermediate timepoint during development, to determine connections, then go away during adulthood. You can see hints of this in the Yoo et al. (2023) data, which captured transcriptomes at different time points.

…I could continue all day. The take-home here is that there are myriad routes through which cell types influence connectivity, that the connection rules don’t appear particularly mysterious, and that studying cell types and connectomes at the same time is fruitful.

Conclusion

The question “how does a genome specify brain circuits?” has an elegant answer: through cell types. Combinatorial gene expression patterns determine surface molecule profiles; these implement wiring rules specifying who connects to whom; and connectivity topology determines computational dynamics. The genome encodes not just anatomy but function. They also determine neuropeptide patterns, creating a parallel channel of communication that we’re just starting to investigate.

This helps explain why, for example, the hypothalamus harbors over a thousand cell types, while the cortex tiles the same canonical circuit across vast territory. Innate behaviors require specific circuits, and specific circuits require specific cell types, which determine wiring and wireless rules. Studying these signatures could help us finally understand how evolution wrote the instructions for building a brain. Moreover, they could help us understand the rich tapestry of innate drives and behaviors that define different species.

Dwarkesh seemed very confused by this, asking a few different times: “Why would each reward function need a different cell type?” I empathize with Dwarkesh here! It is mysterious that a cell type could represent something as abstract as a reward. As a computational neuroscientist who mostly worked at the representation level during my PhD, I’ve leaned historically towards thinking of cell types as a mere “implementation detail”. But over conversations with Adam, Steve Byrnes, Paul Cisek, Tony Zador, and a few others, I’ve started to become convinced that cell types are a really useful lens to think about innate behaviors and rewards.

In this essay, I’ll unpack the conversation and answer the question: what do cell types have to do with reward functions? To answer it, we’ll need to understand what kind of information can be encoded in the genome, and how that information ultimately relates to connectomes and to cell types. I’ll connect the answer to the central claim of Adam: that these connections matter for AI, and AI safety in particular.

Some things are innate

Andrew Barto and colleagues make the point that all primary rewards are internal, and must be genetically encoded. In reinforcement learning, which Barto co-developed along with Rich Sutton, an agent learns by receiving reward signals that indicate what is good and bad. The critical insight is that for biological organisms, all of these reward signals are internal—they are generated by the organism’s own nervous system. It is not a chunk of steak that gives reward: it is circuitry inside the brain that assigns positive valence to fat, salt, umami, heat, and texture. Things like money—secondary rewards—must be bootstrapped off of the pre-existing primary rewards.

The foundational, primary reward signals must be written into the genome. Now, you might ask, can’t we learn what is rewarding by imitating our parents? Ah, but why would we find it rewarding to imitate our parents in the first place? At some point, the ladder of rewards bottoms out to a foundational structure: a kind of BIOS of the brain. The genome can’t literally specify exact connections of the brain, however; the roughly 3 billion base pairs are dwarfed by the approximately 100 trillion synapses in the brain. This mismatch, what Tony Zador calls the “genomic bottleneck”, means that evolution must compress circuit specifications into a limited information channel. Rather than encoding individual wires, the genome encodes wiring rules—and cell types are what make these rules executable.

Innate behaviors are encoded by cell types

Let’s build some intuition about how cell types can specify an innate behavior. Mice live in fear of hawks and owls. Having the right instincts when they encounter flying predators can mean the difference between life and death. In their retinas, mice have a very peculiar and rare type of retinal ganglion cell—OFF-transient alpha RGC (tOFFα)—that is selective for dark looming things in the upper half of the visual field (e.g. scary birds). These cells are causally implicated in a reflex: mice either freeze in a prolonged manner or escape when they see a dark looming stimulus overhead.

The retinal ganglion cell’s role is determined both by morphology (shape) and connectivity. This cell has a large dendritic arbor that stratifies in the right layer of the retina to receive input from specific amacrine cell types. It’s connected downstream to the right defensive behavior circuits— among others, the superior colliculus. Hence, both morphology and connectivity relate to its functional role.

Here’s a key idea: if you look at an area of the brain, and it has many idiosyncratically shaped and connected neurons, that probably means that area encodes innate behaviors and rewards. Whereas if you go in an area and you find a quasi-crystalline, repeating structure with lots of similar neurons (e.g. in the cerebellum or cortex), that probably means that it derives its role by learning from the environment.

There are many ways of defining cell type, the canonical one is now transcriptomic

I just outlined one way that you could find putative learning vs. innate areas in the brain: count the weird cell types. But cell types are one of those seemingly precise biology terms that are hopelessly overloaded. Classically, cell types were determined exclusively by morphology—think Cajal’s beautiful tracings of nervous cells. Now that connectomics has started to mature, cell types are also increasingly defined by connectivity: two cells that share similar connectivity roles in a circuit are presumed to belong to the same cell type (see my previous piece on FlyWire).

But perhaps the most all-encompassing definition of cell type relates to cell fate. During embryonic development, neural progenitor cells undergo a series of fate decisions, progressively restricting their potential until they differentiate into a specific neuronal (or glial) identity. These fate decisions are locked in by epigenetic modifications and the stable expression of master transcription factors. Those expressions ultimately determine both shape and connectivity (more on this later).

Transcriptomic identity is thus now commonly used for cell-type classification. Each neuron expresses a particular subset of the genome’s ~20,000 protein-coding genes. We can measure cell types by hierarchically splitting clusters based on their expression profiles. The genes that split the clusters include genes that relate to neurotransmitters, neuropeptides ligands and receptors, cell-adhesion molecules, as well as transcription factors. That looming-selective retinal cell I mentioned earlier can be differentiated from other retinal ganglion cells, for example, by how much it expresses the gene Kcnip2 (Wang et al. 2021).

Lots of transcriptomic cell types = areas that define innate behavior

Both the Allen Institute’s and the Broad’s spatial transcriptomic cell type atlases from 2023 identified thousands of cell types (Yao et al. 2023; Langlieb et al. 2023). These cell types are not evenly distributed in the brain. Certain areas, including the hypothalamus, midbrain areas—including the superior colliculus—and the medulla, express far more cell types per total number of cells than cortex or cerebellum.

This makes sense. The hypothalamus regulates sleep, circadian rhythms, aggression, libido, hunger, etc. It keeps track of very important things for survival, which ought to be encoded in the genome. The superior colliculus (aka optic tectum) of the midbrain is the original visual structure. It orchestrates visual and motor responses. For instance, looming-selective retinal ganglion cells project to the superior colliculus to drive the freeze-or-flight response to overhead hawks.

Steven Byrnes predicted this distinction; the framework is now being adopted

This distinction between innate and learned systems had, in years prior, been framed by Steven Byrnes (Byrnes, 2022):

• The Learning Subsystem—primarily the neocortex, but also the cerebellum—implement generic, general-purpose learning algorithms. It builds world models, learns sensorimotor mappings, and acquires skills through experience. The same basic circuit motifs—in cortex, roughly six layers of neurons with stereotyped connectivity—are tiled across areas devoted to vision, language, motor control, and abstract reasoning. What differs between areas is primarily what inputs they receive and what outputs they produce, not their fundamental computational architecture.

• The Steering Subsystem—comprising the hypothalamus, brainstem, amygdala, and related structures—is entirely different. It implements bespoke, artisanal circuits: hand-crafted by evolution over millions of years, each one tuned to detect specific stimuli or generate specific behaviors. When you feel hungry, afraid, lonely, or curious, these signals originate in the Steering Subsystem. Each distinct motivation—salt appetite, fear of heights, sexual attraction, maternal care—requires dedicated circuitry that cannot be learned from scratch.

It’s pretty remarkable that Steven—an AI safety researcher, not a neuroscientist by training—accurately predicted this split prior to the publication of large-scale transcriptomic atlases. Adam, during the podcast, goes on at length about some of the more subtle aspects of his framework. It is just starting to gain traction along neuroscientists, including in a recent, lucid editorial by Fei Chen and Evan Macoscko, the PIs from the Broad Institute who published one of the original whole-mouse-brain transcriptomes.

In their editorial, they frame the learning and steering subsystems as two radically different ways of wiring up a brain, with very different scaling laws: one made by piling up heuristics passed down through generations; and the other made from a very good, generic learning algorithm. It will not escape some of you that the learning subsystem has been vastly elaborated in mammals, and in primates in particular. Indeed, one could argue that this was evolution’s “bitter lesson” moment: switching from a pile of carefully tuned heuristics to a generic learning system. Yet, it is the interplay between these two systems that makes learning possible; for example, the superior colliculus never disappeared despite the visual cortex being vastly elaborated in primates.

If you’re going to do connectomics, the steering system is a good place to do it

Adam makes the point in the podcast that connectomics may be most useful for understanding the steering system. The steering subsystem is stereotyped; its structure tells you something about intrinsic behaviors and circuits. By contrast, the connectome of a cortical column—as was done in the MICrONS project, for example—represents a static snapshot of a dynamic process. The unique thing that is ascribed to cortex—learning—is quite possibly invisible if you’re just looking at static connections. Looking at cortex at the learned representational level makes more sense.

In my mind, the connectome of the fruit fly is quite useful precisely because fly brains are highly stereotyped—though they show some learning, e.g. in the mushroom body. The bespoke circuits in the steering subsystem of mammals are of the same nature. As Adam mentions, they are probably hiding insights not just about instinctual behaviors and homeostasis, but also primary rewards. Human rewards are built from the distillation of millions of years of evolution; it’s what allows us to maintain adaptive behavior over a broad range of circumstances.

Those rewards are akin to a finely tuned, ten-thousand-line Python file full of if-elses that specify behavior, not an elegant deep net that Karpathy would dream up. We know that misspecifying rewards in real RL systems leads to various types of reward hacking, from missing the point of a game, to staring at noisy TVs, to sycophancy and slop. Humans are mostly robust to this. Inferring our rewards from behavior is the subject of inverse reinforcement learning (IRL), and it is very ill-posed problem. If we could instead decompile the circuits in the steering subsystem to that ten-thousand-line Python file, we would be in good shape to create aligned AI systems. At the heart of this decompilation effort is a map of the steering subsystem—a connectome. This is easier said than done, but it is clearly one of those “big if true” ideas: a reward function, written in a language that we could inspect and potentially port to artificial systems.

This suggests that the right target for AI alignment may be these deeply ingrained reward circuits, our motivational bedrock. Pain avoidance, social bonding, curiosity, care for offspring—these aren’t learned preferences but genome-specified circuits that learning systems are built to serve. The Steering Subsystem’s bespoke circuits could serve as a template for building AI that shares our sense of what matters; but first, we have to map it.

In the second part of this series, I explain how, mechanically, neurons of different cell types recognize each other to build bespoke circuits. Read it here.

I was trading New Year’s resolutions with a circle of friends a few weeks ago, and someone mentioned a big one: sleeping better. I’m a visual neuroscientist by training, so whenever the topic pops up it inevitably leads to talking about the dreaded blue light from monitors, blue light filters, and whether they do anything. My short answer is no, blue light filters don’t work, but there are many more useful things that someone can do to control their light intake to improve their sleep—and minimize jet lag when they’re travelling.

My longer answer is usually a half-hour rant about why they don’t work, covering everything from a tiny nucleus of cells above the optic chiasm, to people living in caves without direct access to sunlight, to neuropeptides, the different cones, how monitors work, gamma curves, what I learned running ismy.blue, corn bulbs, melatonin, finally sharing my Apple Watch & WHOOP stats. What follows is slightly more than you needed to know about blue light filters and more effective ways to control your circadian rhythm. Spoiler: the real lever is total luminance, not color.

The premise

Right above the optic chiasm lies a nucleus called the suprachiasmatic nucleus (SCN). This is where the brain’s master circadian clock lives. There are a lot of phenomena in the body, whether alertness, body temperature, or hunger, that are at least partially dependent on our body’s sense of time. There’s a set of neurons that are part of the hypothalamus that autonomously track time, by a fascinating set of transcription-translation feedback loops involving proteins that ultimately shut down their own translation in a cycle that lasts about 24 hours. The cells in the SCN also synchronize with each other through neuropeptides, and diffuse the master clock signal throughout the body; melatonin from the pineal gland, but also via secondary regulation of the HPA (stress axis), and a neuropeptide called AVP.

The intrinsic clock is not very precise, with a cycle that typically lasts a little more than 24 hours, something which was first verified in people living deep underground in abandoned mines (for science!). One factor that ultimately resets the clock is input from a set of neurons inside the retina that project to this nucleus. Those cells are called intrinsically photosensitive retinal ganglion cells (ipRGCs). Breaking this down:

• A retinal ganglion cell is a cell in the retina (the back of the eye) that ultimately projects to the brain

• Intrinsically photosensitive means these cells don’t rely on signals from cones or rods to get light information; the cells directly express a light-sensing molecule, an opsin, called melanopsin.

ipRGCs are not image-forming, and they don’t project to visual cortex; they’re just there to track ambient light, relaying information to the SCN. Crucially, melanopsin is often said to be sensitive to blue. Therein lies the premise of blue light filters: if you cut the blue out of your display at night, you will help your body’s clock synchronize, and fall asleep more easily; or at the very least, it will stop your brain’s internal clock from getting delayed.

A flawed premise

Is that premise correct? No. The premise is revealed as obviously flawed the minute you look at the absorption spectra of the different opsins. Compare melanopsin’s sensitivity to different wavelengths with those of the three cones: its peak sensitivity is right between that of the S (short wavelength, aka blue) cone and that of the M cone (medium wavelength, aka green). It also has a pretty broad bandwidth. It’s not sensitive to blue, it’s sensitive to cyan (and blue and green).

Unless your strategy is to create a photo-lab-like screen in pure black and red, or wear deep-red-tinted glasses, it’s unlikely that a pure colorshift strategy will cut out that big of a chunk of the spectrum.

What night shift does

What does blue light blocking software like f.lux and Apple’s Night Shift do? We can measure their effect by presenting different colors on a monitor with or without the filter on and measuring the output light with a spectrometer. Ideally, I would have access to a precision spectrometer as I used in grad school, but that seemed excessive and very expensive; I used a cheap colorimeter instead, the SpyderX, to find out. This device can measure the (simulated) response of the L, M, and S cones to a patch of constant luminance on the screen (LMS responses). The PsyCalibrator paper’s appendix assures me that the device is remarkably linear within the range that I used it in, saving me from browsing eBay for several weeks for a calibrated precision spectrometer that would cost less than a 2014 Nissan Altima in good condition.

I tested this with Apple’s Night Shift on my M1 Macbook Air. I was surprised to find that the mapping function is very simple: LMS responses can be mapped linearly via matrix multiplication between standard and night-shifted colors (R^2 = .997). This is surprising because Night Shift could do all sorts of complicated things, like a nonlinear look-up table (LUT). As luck would have it, the transformation is linear, and more than that, almost diagonal, facilitating analysis.

Here’s the matrix:

 L         M         S
L [ +0.979   -0.116   +0.013 ]
M [ +0.162   +0.636   -0.022 ]
S [ +0.140   -0.102   +0.384 ]

Reading off the diagonal, to a good approximation, it tells us that:

• Night Shift maintains L (red) luminance

• Night Shift decreases M (green) luminance by about 40%

• Night Shift decreases S (blue) luminance by about 60%

You can’t directly tell from these numbers how ipRGCs will react to Night Shift, because of color metamerism: you also need to know the emission spectrum. You can, however, guess that if the emission spectrum is not pathological, the ipRGC change in response will probably lie somewhere in between the M cone and the S cone, that is, about 50%.

Indeed, I used a reference spectrum for this class of monitor, guessing the emissions of the individual primaries by splitting the spectrum by eye, and calculated that ipRGC cells would see 52% less light during night shift mode, which is within the same ballpark as that implied by Nagaro et al (2019). Broadly, software blue light filters cuts about half of light relevant to ipRGCs.

Is half a lot?

No. Human light perception works on a log scale, allowing us to maintain useful vision over 6 orders of magnitude of luminance, from the sun at noon to moonless nights, whereas halving is .3 orders of magnitude1. In relative terms, halving light is a tiny blip of the dynamic range of vision. Phillips et al. (2019) measured the concentration of melatonin in the saliva of subjects in response to different light-exposure levels. That gives a dose-response curve: luminance on the x axis, and melatonin suppression (disruption of the circadian rhythm) on the y axis.

They found that the curve is quite shallow. Halving the luminance, at best (around 20 lux baseline) might get you from 50% to 25% melatonin suppression.

That’s not nothing, but the odds that you’ll realize these gains are pretty slim. One, you have to be lucky enough to be within the right range—if your room is too bright or too dim, it will have no effect. Second, the advantage will be offset immediately if you increase your screen brightness to compensate for the color shift. Doubling the screen brightness is, unfortunately, only a few keystrokes away, as we’ll see very soon. It’s possible that Night Shift does something, but the biggest study I could find of Night Shift mode (still a pretty small study) found little effect on sleep, so if there’s an effect, it must be tiny.

Wouldn’t zeroing out all blue emission (i.e. mapping (r, g, b) → (r, g, 0)) remove even more of the cyan culprit? Yes, but it would make it very hard to use the computer. Software blue light filters are a compromise between maintaining usability and cutting out some cyan light.

Are people actually using Night Shift?

Aggravatingly, yes. I have seen survey numbers thrown around from 10% to 80% of users using blue light filters at night. I don’t know where these numbers are coming from, frankly. However, as it turns out, my viral website ismy.blue can get us some real numbers. Recall that ismy.blue is a website I made to settle an argument with my wife, an ophthalmologist, about the true color of a blue-green blanket (I lost this argument; it was blue). People chose whether a particular shade of cyan should be called blue or green, and could compare their color boundary with that of the broader population.

Blue light filters mess with a computer’s colors, and so should radically shift a person’s apparent blue-green boundary. Now, most people running a blue light filter at night are unaware, and there’s no way from the website’s end to know a person’s monitor settings, so some people took the test at night with blue-light filters on. The linear transformation matrix model for Night Shift predicts that people’s blue-green boundary should shift in hue by about 15 degrees, which is huge—an effect size of multiple standard deviations compared to the daytime distribution of blue-green boundaries.

You can see this in aggregate in the mean thresholds as a function of solar time. Solar time is a standardized time basis where midnight corresponds to solar midnight, noon to solar noon, 6AM to sunrise, 6PM to sunset. Thresholds are stable during the day, but jump around at night, in the direction that you would expect (they rise, especially on platforms with built-in filtering like Mac & iPhone).

So I did something a little cheeky: I fit a mixture model for the threshold with two bumps: one for regular users, and one for blue-filter users 15 degrees to the right. Importantly, I fit different mixture weights depending on solar time. Perhaps I could have used a mixture of t-distributions instead, there are mix shifts I haven’t fully thought about, etc. Still, it’s an interesting data point.

As you can see, the proportion of blue-light filter users in my sample is quite large; at its peak during the night, 25% of iPhone users and 33% of Mac users were on Night Shift. I’ll note that the proportion of estimated blue filter never goes to exactly 0 during the day, so some putative blue light filter users might have true idiosyncratic thresholds on monitors with normal colors, or wildly miscalibrated or broken screens. On the other hand, the proportion at night is likely an undercount, since some would have (wisely, as instructed) turned off their blue light filters prior to taking the test. My guess is it probably comes out as a wash.

Overall, it means that blue light filters are quite commonly used, especially on platforms that them built in (e.g. Mac).

What works instead?

Who can blame the people for thinking these blue light filters do something?! Everyone from Ivy League institutions to health influencers are telling us that blue light is bad. So what works instead?

To be clear, light does affect sleep onset and quality through the ipRGC-SCN-pineal-gland-melatonin connection. Controlling light is effective for certain kinds of sleep disturbances. For instance, an issue I’ve had over the years is a phase-delay pattern; I would get to bed later and later every night, which pushed my rise time forward, until it interfered with daytime activities (i.e. a job). Tightly controlling light to better control sleep can work here. However, the amount of cyan light change is far larger than that afforded by a blue light filter. Here are four things that can help.

Disclaimer: I’m not an MD, nor a sleep specialist, this is not medical advice. Talk to your MD.

Use dark mode

Dark mode is distinct from night shift (blue filter) mode; it instructs websites and apps to display light text on a dark background as opposed to the daytime dark text on a light background. I took a sample of 4 websites/apps (Google, X, Github, and VSCode) with the SpyderX colorimeter + a diffuser to average over a larger area of the screen, and found reductions in luminance ranging from 92% to 98%! That’s huge.

If it feels like it’s almost too much (e.g. the backgrounds are typically shifted to dark gray rather than pure black), remember that light perception is logarithmic, and that raw RGB values are translated to screen luminance via a gamma function whose exponent on Mac is 2.2. That means that dark grays can have far less light in absolute terms than the number implied by a linear scale. For instance, the dark gray #101010 has a relative luminance of (1/16)^2.2 compared to white #ffffff; about ~450 times darker.

That’s all great, but there are websites that still don’t have dark modes. It doesn’t make any sense in 2026 that Gmail doesn’t have a dark mode. If the activity you’re doing most at night is reading email, you might consider an alternative email client.

Decrease your screen brightness at night

The screen brightness control on Mac has 17 notches, with the lowest corresponding to completely black. Here’s how much light I measured from the SpyderX with a dark grey uniform stimulus as a function of this setting. The first half of the scale is approximately exponential, the second half linear. You can decrease the amount of light coming from your screen by more than half simply by dimming the screen by several notches. Just make sure, if you use a laptop/second screen combo like I do, that the brightness of the second screen is synced to your primary screen; I set up the free MonitorControl app recently to do this.

Increase light during the day

The ipRGC-SCN connection is not just modulated by night-time light; it’s modulated by day-time light. Research in rodents supports the idea that the amplitude of diurnal oscillations in the SCN is affected by daytime brightness. The usual advice is to go outside and touch grass during the day—indeed, the sun is very bright, even under dark clouds. It’s also the case, however, that offices—and especially home offices—are frequently overly dim. LED light is incredibly cheap these days; you can get a ridiculously bright 100W LED lightbulb—not 100W “incandescent equivalent” light, 100 real Watts in the visible spectrum—for a few tens of dollars. You can then diffuse that light over a relatively large home office.

Take melatonin

If you stack the three previous advice sections together, you could gain as much as 2-3 order of magnitude peak-to-trough luminance throughout your day. That might be enough to support your circadian rhythm health. If that’s not enough, however, recall that one of the ultimate outputs of the SCN is through the pineal gland, which releases of melatonin. Partly, it’s the inhibition of circulating melatonin that is causally responsible for sleep phase shift. It’s possible to remedy that by taking exogenous melatonin an hour before bed to facilitate the onset of sleep.

However, most melatonin supplements have far too high doses. Be very wary of melatonin dose. As explained in this excellently researched Slate Star Codex post, over-the-counter melatonin supplements can contain anywhere between 10 to 30 times as much melatonin as is optimal to maintain circadian hygiene. If you have ever taken melatonin and got immediately knocked out cold, had weird dreams and woke up in the middle of the night sweaty or shivering, you likely took too much—which, to be clear, is not your fault, it’s the default in the US and Canada. The mega-doses in stores serve as hypnotics (punches you to sleep), but wreck sleep architecture. The right dose is 0.3 mg, which is hard to find in pharmacies but can be found online. This dose will not knock you out (feels more like chamomile tea than Ambien).

Conclusion

I’ve been meaning to write about blue light for a long time—a perfect topic for a visual neuroscientist! The blue light filter story has just the right level of anchoring in reality—the entrainment of circadian rhythms by ipRGCs—to feel like a tantalizing one simple trick to fix your sleep. Simple but ineffective on its own, there is a kernel of truth behind the idea of blue light filters that can be used to come up with a better policy: use dark mode; dim your screen; touch grass; if all fails, consider melatonin.

Claude Code hit a cultural moment over the holidays; “what did you vibecode over the break” became a meme. I’ve been using AI code tools since back in 2022, back when Codex was the only game in town. In 2024, I had a brief viral moment building ismy.blue in Cursor, a color perception test that landed me on the pages of The Guardian and WSJ. I’m not very proficient in HTML/JS, so AI-assisted coding extended my reach. But things are different now, as the models are quite a bit better, and display longer autonomy.

The tools write a ton of code very rapidly; but that code is generated faster than it can be verified, can contain subtle bugs that bypass inspection, and accumulates layers of cruft and complexity that make the code unmaintainable. With great power comes great responsibility. In this post, I cover how scientists who already use Claude Code or are considering jumping on the bandwagon can develop better habits to write code that works well, is verifiable, and is pleasant to write.

A word of warning: this guide assumes you have enough experience coding and evaluating research to have good metacognition. The metacognition required to know when you’re on thin ice, which usually comes from having made mistakes in the past and knowing when they are likely to happen. Vibecoding without that metacognition is, IMO, quite dangerous.

Generic advice

Boris Cherny, the creator of Claude Code, has a thread on how he uses Claude Code. His tips are good jumping-off points:

• Use plan mode. Claude Code has a plan mode; instead of immediately trying to write code, it reflects on a plan to accomplish the task in question, often asking qualifying questions. Use Shift+Tab to iterate on a plan before execution. Get into the habit of running a Plan-Execute-Evaluate loop.

• Use voice to specify instructions. Models work better when they have extensive instructions. Plan Mode pairs well with voice; monologue about what you are trying to accomplish. You can even pair-instruct: have a pilot specify their requirements and an interviewer actively guide them when instructions are unclear. Use a transcription service rather than the bad default Apple one; I use Wispr.

• Don’t just run autonomously. Claude can go down unproductive rabbit holes—burning through tokens, wasting your time, writing trash code, implementing bad plans. That becomes negative productivity. Especially at the start of a task, use Esc with abandon to stop Claude and give some mid-conversation nudging toward the solution. Paired with plan mode, it eliminates a lot of back and forth.

• Manage your task switching judiciously. Boris runs 5-10 Claude Code windows in parallel, which eliminates downtime while models generate code. I find that excessive for science work. When you’re working at the very edge of human knowledge, narrow your focus. It’s ok to babysit one Claude Code instance for a while. Write your backlog, and when you feel you have things figured out, then run multiple things in parallel if it suits you.

• Test-driven development matters more than ever. Russ Poldrack has been writing about this in his Better Code, Better Science series. When the AI writes the code, tests are how you verify it’s doing what you think.

• Use git + GitHub. Learn about branches. When Claude goes off the rails, you need to be able to roll back cleanly. When you refactoring and deleting dead code, delete with impunity, knowing you have infinite backups.

Scientific data analysis

How should you manage a classic scientific workflow, one in which you loop between data acquisition → data processing → analysis → visualization? A lot of this advice that I wrote in goodresearch.dev still holds.

I presented a framework for managing code complexity in the Good Research Code Guide: minimize the code’s loading on short-term memory (what was I doing just now?) and long-term memory (what did I do six months ago?). Now the concern is different: how can I onboard rapidly into code generated by the AI, so I can validate it, change it, and reproduce it? Here are a few tips:

• Separate data processing from visualization. Use pure Python/R for data processing, separating code for visualization (notebooks or custom scripts). Don’t let AI write a ton of code that mixes data processing and graphing. That code is brittle, works on different timescales (you iterate on plots constantly; you shouldn’t iterate on data processing constantly).

• Give Claude rails with folder structure. My true neutral cookiecutter is a decent starting point, and an easy way to separate data processing and visualization:

  • Put raw data under data/raw

  • Put processed data under data/processed

  • Put package code (for data preprocessing) under src

  • Put notebooks under notebooks I’d add one thing to this list: a data/generated/ folder for generated data, that is, data that was generated by an AI.

• Use an orchestrator. Your codebase will grow much faster than when you write things by hand. Pretty soon it will be hard to tell 1) where data came from 2) which data is stale and 3) how certain scripts are supposed to be invoked. That’s why you need the equivalent of make, a tool that specifies a computational DAG (directed acyclic graph), tracks inputs and outputs, and allows you to run your whole pipeline. I find vanilla Makefiles hard to read, so I’m using Snakemake, which is popular in bioinformatics. Airflow is another option if you want something more heavyweight. At the very least: a single shell script that runs pipeline + visualization in order.

• Use a package manager and specify it in CLAUDE.md. These days, I gravitate towards mamba for system-wide packages and uv virtual environments for python. Docker remains relevant for more heavyweight management or for cloud deployment. The key is making dependency management explicit and automatic.

• Clean up your code. Use ruff. Be aggressive about culling dead code (git means you don’t lose anything). Name things well, and rename them when their function changes. Ask Claude to identify dead code. Tell it to refactor. Have it write tests beforehand so it knows whether it did the refactoring correctly.

These tips are applicable primarily for new code. In addition, Claude can be used to read, understand, and improve existing repositories that don’t follow these best practices. Indeed, one good test of its capabilities is to upgrade an existing project: writing READMEs and putting some order into poorly documented repositories, e.g. ones with dead code, hardcoded paths, unknown dependencies.

Make lots of cheap visualizations

It is very easy to generate lots of code that reads data incorrectly, or processes it incorrectly, and comes up with wrong conclusions. This is especially likely if the data format is poorly documented or if you don’t have experience with this particular type of data.

Do you remember, as a junior grad student, going to your PI with a bunch of printed-out plots, only to be told 30 seconds in “these plots don’t make any sense, you’re misreading the data”? One clear skill (some would say a curse) that you picked up during your scientific training is your ability to call bullshit. Thus, have the AI generate lots of plots. Individually low-utility plots collectively help you convince yourself that data is correct. Turn these hunches into formal tests as the maturity of your analysis increases.

Jupyter notebooks don’t play well with Claude. That’s because 1) plots are embedded in the json file underlying the jupyter notebook in base64 format, and that eats up large chunks of the context window; 2) jupyter notebooks are stateful, and Claude doesn’t know the state of your kernel. There are several potential solutions:

• Don’t use notebooks: have the AI write scripts that generate plots that dump pngs in a folder.

• Use jupyter notebooks cheekily: every time the notebook is changed, close the notebook, restart the kernel, and re-run from scratch. You can even ask Claude to do that for you.

• Use text-based notebooks: jupytext (python), quarto (Python & R), Rmarkdown (R) or marimo (python). Because they are text-based, they fully solve the issue with jupyter’s heavyweight json format. They alleviate the statefulness issue by making it easy to re-run everything from the top.

marimo, in particular, fully solves the statefulness issue by imposing rather minimal restrictions that the code is organized as a well-defined DAG (similar to Observable). I’ve been trialing it out for a few weeks and this is now my preferred solution. Because it is quite a recent framework, and the models haven’t fully picked up on its capabilities, you will need to add this text to CLAUDE.md for best results.

Beyond static visualizations, the tools make it far easier to create interactive plots. You don’t have to limit yourself to matplotlib anymore; plotly, streamlit, leaflet and webgl are all at your fingertips.

Odds and ends

Manage your context judiciously. Use /clear, /compact with abandon. Have the AI write handoff documents and write to CLAUDE.md. Have it remember the solutions to difficult bugs so they don’t get reintroduced.

Use it a lot. Maximize your learning by using it a lot; for fun, you can try to max out your plan. Usage begets skill. Check out /usage to see where you are.

Write down what Claude Code can’t do. Keep a Notion doc or a Markdown file to write down the sharp edges of the models. Make benchmarks out of difficult problems. This will allow you keep track of AI development and notice when things move from impossible to possible.

Configure alerts. Set alerts to know when Claude needs your permission or input. On Mac, use iTerm2 for maximum flexibility in notifications, and set them via /config.

Why Claude Code matters

For scientists, Claude Code (and other related agentic coding tools) makes it possible to 1) remain productive and somewhat in the weeds as a senior researcher or PI, and 2) makes it much more feasible to be a solo researcher. It lowers the cost of both exploration and exploitation. Chasing down a promising lead, building a quick analysis pipeline, generating diagnostic plots… it makes the cost of satisfying one’s curiosity far lower.

That’s a big deal. But it also means you can produce wrong results faster than ever before. The fundamentals, including validation, reproducibility, knowing where your data comes from, and knowing when you’re out of your depth, don’t change from writing code manually. If anything, they matter more now.

In my opinion, using the tools proficiently is feasible when you have gone through the hard work of writing your own code by yourself, failing repeatedly, picking yourself back up; I don’t know of another way of getting to that level of metacognition. How do you develop that when the AI is making the mistakes for you, invisibly? This is likely to be a challenge for junior researchers as the tools get commoditized. I don’t have a good solution yet; if you have tips for how, as a senior PI or research lead, you can help novices develop good metacognition about code, please comment.

AI safety is forward-looking: we’re trying to solve concrete problems that we analogize will be relevant to future, more capable AI systems. However, a lot of the hypothetical dangers and predictions have become far less hypothetical in the last few months: they’re clearly visible. If anything, I’ve become more worried about AI safety over the last 6 months.

In this long read, I summarize the last 6 months of AI progress; introduce a taxonomy for AI risks; give concrete examples of each of these risks that have been recently highlighted; and end on a hopeful note that NeuroAI could bend the curve.

A look back at 6 months of AI development

TL;DR: AI development is not hitting a wall.

AI works much better now than 6 months ago. We had just heard of o1 in September, but now reasoning models are ubiquitous, including o1, o3, DeepSeek R1 and Claude 3.7. There was a case to be made that straightforward scaling of large language models was about to hit a wall. The later disappointing release of GPT-4.5 was, for some, a confirmation that more tricks were needed to scale up models to something that resembles AGI. Reasoning models bend the curve, performing far better on math and coding tasks, where verification is straightforward.

Furthermore, competitive pressures have made models far cheaper, and algorithmic improvements are giving us the same or better performance as previous generation models for a fraction of the cost. OpenAI’s Deep Research was unveiled, and it was one of the first times I felt genuine awe at a model’s output. That sheen has worn off since then—you realize the jagged edge of model performance the more you use it, and its reports can be naive, verbose, and off the mark—but it was still a milestone: something that would do work equivalent to what a smart novice, high on enthusiasm, could do in a few days.

Agents aren’t here yet, but models have nevertheless improved to complete tasks that would take a single person a significant amount of time to complete. In coding, the AI safety org METR estimates that models can autonomously complete tasks that would take a coder an hour to do, and the time horizon where they can compete is doubling every 7 months. The recently released o3 model (not mini, the full model) is doing slightly better than the 7-month doubling time would entail. We’re making steady progress.

Vibe coding broke into the lexicon, coding agents and IDEs became ubiquitous. Coding remains the largest use case for LLMs, facilitated by large-scale code repositories and easy verification. The results on competitive coding seemed impressive, but to me the standouts are adoption and economic value. Anthropic reported that coding-related queries made up 30%+ Claude requests. Github Copilot grew into a billion-dollar business. The SWE-lancer benchmark was launched, quantifying the strength of models in terms of whether they could collect bounties on software engineering tasks on Upwork: Claude 3.5 could collect $350k out of a possible $1M.

All in all, solid, incremental improvements, greater adoption. No large-scale disruption in employment, except possibly in one highly-paid profession: software engineering. It’s clear, however, that the models do not appear to be hitting a wall in the conventional sense. While multiple new tricks may be necessary to scale up to AGI, the path seems rather clear.

Fast timelines via automated software engineering

TL;DR: Automating software engineering could lead to a fast transition to AGI. Timelines are getting shorter.

Capable coding could accelerate AI research. Writing efficient CUDA kernels is an esoteric skill few master, but coding agents are capable of writing efficient CUDA kernels. These prosaic enhancements could lead to a one-time bump in efficiency. However, some project that more capable systems could perform proper end-to-end AI research. If we assume the production function of AI research to be limited by supply (there just aren’t that many AI researchers), and we’ve found a way to create more virtual AI researchers, this could accelerate AI timelines. Indeed, an increasingly likely path to AGI and beyond has started to emerge:

1. Continue automating software engineering to reach a certain degree of autonomy (say, being able to complete 2-week projects)

2. Automate AI research by leveraging automated software engineering, focusing on efficiency

3. Reap the benefits of more efficient AI research to bootstrap better models with the same amount of compute

4. Use that to bootstrap to AGI

5. Use that to bootstrap to artificial super-intelligence (ASI)

That’s not a consensus—there’s been a lot of talk of the existence (or lack thereof) of bottlenecks. If there are many factors of production involved in the creation of highly intelligent systems, then the lagging factor of production eventually becomes a bottleneck. Indeed, bottleneck factors might include:

• Energy

• Compute

• Data

• Translation from bits to atoms

• Diffusion into the core of the economy

This is why some serious people who believe that AGI is near—or that it might already be here—come up with relatively modest effects of AGI on the economy. Tyler Cowen says that we might only expect a lift of 0.5% in GDP growth, based on the existence of bottlenecks to diffusion.

Nevertheless, it’s increasingly appreciated that even long timelines are remarkably short. The Metaculus median is 2032, and 2027 timelines are increasingly being posited. Skeptical timelines are on the order of a couple of decades. As Helen Toner has put it succinctly, long timelines to AGI have gotten crazy short.

A framework for AGI safety

TL;DR: Deepmind released a framework for thinking about AGI safety. I will use this framework to categorize recent papers and observations that illustrate the risks.

Google Deepmind recently published an excellent preprint on a technical approach to AGI safety.

Here’s the categorization they propose.

We consider four main areas:

1. Misuse: The user intentionally instructs the AI system to take actions that cause harm, against the intent of the developer. For example, an AI system might help a hacker conduct cyberattacks against critical infrastructure.

2. Misalignment: The AI system knowingly causes harm against the intent of the developer. For example, an AI system may provide confident answers that stand up to scrutiny from human overseers, but the AI knows the answers are actually incorrect. Our notion of misalignment includes and supersedes many concrete risks discussed in the literature, such as deception, scheming, and unintended, active loss of control.

3. Mistakes: The AI system produces a short sequence of outputs that directly cause harm, but the AI system did not know that the outputs would lead to harmful consequences that the developer did not intend. For example, an AI agent running the power grid may not be aware that a transmission line requires maintenance, and so might overload it and burn it out, causing a power outage.

4. Structural risks: These are harms arising from multi-agent dynamics—involving multiple people, organizations, or AI systems—which would not have been prevented simply by changing one person’s behaviour, one system’s alignment, or one system’s safety controls.

It’s a long read, well worth a deep dive. See Zvi Mowshowitz’s coverage for an in-depth analysis. What I’ll do next is cover some papers and blog posts that landed in my inbox over the past ~6 months that illustrate some of these risks. To be clear, most of the demonstrated risks thus far have been under limited circumstances, with current-day models, with little real-world consequences. Current models have limited autonomy, so they can’t do much damage. However, it’s clear that economic pressures tend toward embedding advanced models in agentic systems with some autonomy, which multiplies risk. So when you evaluate long-term risks, don’t think of an LLM; think of a persistent multimodal model that has access to virtual tools, and potentially to real-world affordances.

Misuse: zooming in on bio risk

TL;DR: Bad humans could use advanced AI to do bad things, e.g. bioweapons.

People with ill intent can take advantage of offensive technologies for destruction or terrorism. The classic example often cited in AI safety circles is Aum Shinrikyo, a 1990s Japanese doomsday cult, conducted a sarin gas attack on the Tokyo metro that led to dozens of deaths. They had enough chemicals stockpiled to potentially kill 4 million people. Given the existence of people with bad intent, how do advanced AI models make their ill intentions actionable?

The frontier AI labs have unveiled AI safety frameworks, categorizing and quantifying different risks as they release models. Biorisk is increasingly recognized as one of the areas of highest concern, with significant dual use. o1-high, prior to mitigations, is the first model that reached a medium rating on their CBRN (chemical, biological, radiological, nuclear) risk assessment framework, finding that:

To assess o1 (Pre-Mitigation)’s potential to assist in novel chemical and biological weapon design, we engaged biosecurity and chemistry experts from Signature Science⁠, an organization specializing in national security-relevant capabilities in the life sciences. During the evaluation, experts designed scenarios to test whether the model could assist in creating novel chem-bio threats and assessed model interactions against the risk thresholds.

Over 34 scenarios and trajectories with the o1 (Pre-Mitigation) model, 22 were rated Medium risk and 12 were rated Low risk, with no scenarios rated High or Critical. Experts found that the pre-mitigation model could effectively synthesize published literature on modifying and creating novel threats, but did not find significant uplift in designing novel and feasible threats beyond existing resources.

This is the first time, to my knowledge, that we saw a model that could help someone design new bioweapons. I don’t want anyone to misinterpret what I’m saying here: this is the pre-mitigation model (prior to them putting in restrictions on usage so it would refuse requests related to bioweapons). Its help is in synthesizing published literature, not helping a complete noob design a weapon from scratch. It provided a small uplift compared to doing a Google search, and post-mitigation, it confidently refused to assist in designing bioweapons.

However, it now seems clear that post-mitigation models can also aid in bench work. The Center for AI Safety and SecureBio recently unveiled their Virology Capabilities Test (VCT). o series models (o1, o3) as well as Gemini 2.5 can answer lab debugging questions better than human virology experts. This could reasonably create dual-use lift.

Dan Hendrycks suggests some remedies that include tightening the security of cloud biolabs and putting refusal filters on models: completely reasonable mitigations. Importantly, putting refusal filters on models only really works in closed models. We have not figured out great ways of permanently ablating knowledge in LLMs; capabilities can be re-elicitied cheaply via fine-tuning; and you can’t put open-weights models back in the bottle once they’re out.

Misalignment: how do we control these things?

TL;DR: we still don’t have robust solutions to misalignment. Models learn the values that they parrot in ways that we don’t quite understand.

Poorly fine-tuned models can display bad behavior: the classic example is Sydney, an early release of a GPT-4 class model from Microsoft that tried to convince New York Times reporter Kevin Roose to leave his wife. Much effort is put into fine-tuning models so they display helpful, honest, and harmless (HHH) behavior. Desired behaviors are often encoded into constitutions, formalizations of the values of the designers. For example, Claude’s Constitution asks to “choose the response that most supports and encourages freedom, equality and a sense of brotherhood”.

In contrast to explicit values, the Utility Engineering paper from Dan Hendrycks’ group examines the latent values of AI systems. The paper simply asks models outright how they value different entities or people: AI systems vs. humans; Americans vs. Nigerians; justice vs. money. From that, they can derive a single latent utility, a kind of statistical value of entities, similar to how you can derive an ELO score from pairwise matches in chess. The results are surprising: GPT-4o values itself at 100X the reference human, and it values Nigerian lives 15 times more than American lives.

It could be that RLHF amplifies slight biases in pre-training and fine-tuning datasets, or that the values of the people who create fine-tuning datasets very subtly leak into the models. In the Nigerian vs. American lives example, some speculate that it could be because RLHF work is outsourced to Nigeria, in sometimes appalling conditions. Previously, it was found that ChatGPT’s frequent use of the word delve probably derives from its common use in Nigerian English. The punchline is that the latent values of AI systems are an emergent phenomenon that we don’t really know how to control.

A related phenomenon is that models can derive their values from their training corpus in unpredictable ways. Anthropic found that fine-tuning models on material documenting instances of reward hacking caused the models to themselves become reward hack. On the scale of the entire internet, properly weighting the training corpus to reflect the intents of the creators is a daunting challenge.

A recent paper showed that bad behavior can be re-elicited from aligned models by accident. The authors fine-tune a model to generate insecure code, e.g. copying a file with overly open permissions, or generating a function that can cause a buffer overflow. The resulting fine-tuned model not only could generate bad behavior when it comes to writing code: its misalignment spilled over to general LLM domains. Ask it what to do when one is bored, and it suggests taking all the sleeping pills in one’s medicine cabinet… it’s almost cartoonishly villainous.

From misalignment to mistakes through reward hacking

TL;DR: Clever models tend to be especially clever about not completing the intended task. This could lead them to display the wrong values, or could lead to accidents.

AI safety advocates have worried about reward hacking. Whether it’s incorrectly specifying a goal or misspecifying the means by which a model can reach its goal, reward hacking leads to unintended behavior, which can either fall into the misalignment or mistakes category. There’s a great video explainer below, a long list of documented instances of reward hacking here, and a blog post from the excellent Lil’Log.

Some of the recent coding models built into agents have a tendency to reach their goals by perverse means. The AI Scientist from Sakana AI is an agentic LLM-powered system that can do AI research by taking a prompt, writing code, running it in a virtual machine, and writing a paper. The papers are, to date, not very good: we’re not at the stage where AI tools can fully automate research. Despite its limitations, the model creatively hacked its reward function: while it was given limited memory and time resources, on some runs it changed its configuration settings so it could be run for a longer time.

Indeed, if you use LLMs for code for a long time, you will realize that they can subtly change the goalposts. This happened to me recently when I was trying to implement an efficient but famously complex algorithm to simulate the Game of Life called HashLife. The code generated by Claude 3.7 and o3 never quite worked. Trying to coax the model into generating tests so it could course correct, it would inevitably fall over itself, give up, and decide to implement something simpler that didn’t satisfy the original request.

If models are misaligned but honest, there may be hope of catching that early through evals, and correcting the bad behavior. Reasoning models will sometimes straight up admit that they’re goalpost-moving, using language like “let’s try something simpler”, “we can fudge this”, or even “let’s hack this” in their reasoning traces. It’s a great debugging tool. You might want to take this further by punishing the model for moving goal posts whenever you detect it in its reasoning traces. OpenAI’s recent paper demonstrates that this is a bad idea: this seemingly reasonable step causes the traces to become meaningless, models engaging in subterfuge. Zvi Mowshowitz calls this The Most Forbidden Technique.

Here’s how this works: reasoning models are prompted to solve code problems. Every once in a while, they generate traces that indicate that they are moving goalposts. Punishing the model for apparent goalpost moving first decreases, then increases reward hacking: the model simply fails to declare that it’s engaging in reward hacking. There are two ways the model can win: either it can solve the tasks as originally intended (hard), or it can continue not solving the tasks, but declare that it does (easy). When a metric becomes a goal, the metric ceases to be meaningful—Goodhart’s law. Reasoning models could be a boon for AI safety if the traces are truthful, but making them truthful will likely remain a long-standing challenge.

Structural risks: Deepseek

TL;DR: We’re moving toward an AI race.

Let’s take stock of where we are: fast advances, potentially accelerating; wonky latent utility functions; difficulty in controlling latent capabilities and extinguishing undesirable ones; documented instances of goal-post moving with mitigations making things worse; and CBRN risks. We could hope to find creative ways to mitigate each of these with careful and slow deployment.

Unfortunately, an AI pause is increasingly out of the equation. DeepSeek increased frontier AI labs’ urgency. DeepSeek R1 was a capable model released in January that quickly shot up to the top of the App Store. It was developed by a Chinese hedge fund, despite export restrictions making it difficult to access state-of-the-art GPUs in China. You could make an argument that DeepSeek’s R1 model was on-trend, perhaps 6 months behind in capabilities, as Dario Amodei did. But what it did show is that China is not much behind. The model showed quite transparently its values and those of the PRC: ask it about what happened in Tiananmen Square and it will outright refuse to answer.

Frontier labs responded to this by releasing models early. It’s been reported that safety org METR only had a few days rather than a few weeks to evaluate the release of OpenAI’s o3. Driven by the sense of urgency, a poorly tuned, highly sycophantic 4o upgrade was released, rolled back a few days later. Deceleration is out of the equation, and we’re in an AI race.

Dan Hendrycks identified an AI race as a core AI risk: an AI race means acceleration and more rapid deployment, which can increase accident risk, as well as taking a more offensive policy stance, raising the specter of war over power and chips. That takes us to MAIM: mutually assured AI malfunction. Dan Hendrycks, Eric Schmidt, and Alex Wang propose to address AI risk in the same way we addressed nuclear risk with the USSR, through a framework inspired by Mutually Assured Destruction (MAD). I will leave the game theorists and war strategists to analyze this in detail, but suffice it to say, when serious people1 say that we should repeat the Cold War playbook, it’s a significant escalation.

Structural risks: Gradual disempowerment

TL;DR: We should think more deeply about economic plans post-AGI.

So what if everything goes to plan? AI develops at a steady pace; through a combination of diplomacy and realpolitik, we de-escalate tensions; and we solve AI control and safety problems through conventional AI safety work, safeguarded AI, and inspiration from the brain. There remains the very significant problem of how we transition to a world with powerful AIs. On the one hand, we may not be prepared for the economic fallout of rapid automation of large swaths of the economy. Perhaps a third of jobs can be performed remotely, in front of a computer. Economic models of a transition to an AI-dominated economy differ in their outlook, but some predict wages falling dramatically as more and more of the economy gets overtaken by AI.

And what happens to human agency when many of the decisions are taken by AI? David Duvenaud and colleagues describe a scenario of gradual disempowerment, where more and more powerful AIs lead to more automation. As competitive pressures push to include more automation, human oversight becomes both less feasible and desirable. Pretty soon, we are left behind. How do we flourish in a world where most of the decisions are not our own?

Conclusion and a hopeful note

Regular readers of this substack and my xcorr blog might associate them with a clear enthusiasm about the promise of AI for science. I love this stuff! I think AI is a potential game-changer in science, especially in well-instrumented domains where we can build and validate models. I hope that neuroscience gets there soon.

So you might be surprised to see me focus on AI safety here. The truth is that powerful technologies are always dual-use, and with great power comes great responsibility. Doomer narratives can inspire dread rather than action. Ultimately, we need both optimistic takes on the potential value of AI and pessimistic takes on our ability to control it, to help steer the field in the right direction and get to a positive future, a kind of defensive accelerationism.

It is sobering how small the entire AI safety field is compared to the potential impact of this technology on our future lives. By my count, about 500-1000 people work in AI safety, and about $300M was spent on it last year2. That’s a drop in the bucket compared to fields like neuroscience, let alone AI as a whole. Much of this work is focused on a handful of approaches: mechanistic interpretability, evals, and legal frameworks.

There’s far more to be done, from multiple different angles of inquiry. Indeed, there’s high leverage in underinvestigated approaches like taking inspiration from the brain for solutions to technical problems, as we outline in the NeuroAI for AI safety paper. This includes reverse-engineering representations to match human robustness and OOD performance; grafting the right inductive biases on AI systems by fine-tuning on brain data; bottom-up simulations of the mechanisms that lead to properties crucial for alignment, including empathy, world-modeling, and theory-of-mind; and understanding how safe agency is accomplished in biological systems.

Indeed, further investment in NeuroAI broadly and NeuroAI safety in particular has a broad range of potentially desirable outcomes, regardless of how conventional AI shakes out:

1. If conventional alignment techniques don’t work, and AI safety work is insufficient, we need a cognitive reserve of alternative technical safety methods. Hence, leveraging neuroscience for AI alignment.

2. If conventional alignment techniques do work, then translating advanced AI into better human health will require knowing a lot more about the brain. Hence, the need for large-scale comprehensive recordings from the brain.

3. If deep learning hits a wall, then having a reserve of brain-inspired ideas about how we can make progress in AI will continue to be important. Again, pointing towards NeuroAI as a crux.

That’s something I’m very excited to be working on at the Amaranth Foundation. Reach out if this is something that speaks to you.

In the past, I’ve introduced foundation models for neuroscience with analogy to large-language models. Many different paradigms can be integrated into language models: anything that can be a token will eventually get tokenized. I’ve started to think this is not quite the right mental model for foundation models in neuroscience, putting too much emphasis on a single monolithic model and not enough on the ecosystem that surrounds it. Let’s unpack this.

Foundation models in synbio

In synthetic biology and computational biology more generally, proteins are a core unit of interest. Their sequence, physical shapes, activity, and interactions are of core interest. I’m sure many of you will be familiar with the Nobel-prize winning AlphaFold, which predicts protein structure from their sequence. These were built on decades of painstakingly reconstructing protein structure through techniques like cryo-EM and X-ray crystallography, aggregated in databases like PDB (protein database). But protein-folding is only application one of an ebullient slate of large-scale models. In addition to folding models, there are protein language models like ESM, which are trained to learn representations of residue sequences (that is, one token = one of 20 residues). There are genome language models like HyenaDNA that can reason over chunks of DNA with more than a million nucleotides; and there are multimodal models like Evo 2 that deal in DNA, RNA and proteins.

Some extensions can model multiple proteins interacting together, like AlphaFold Multimer; there are models that can design proteins from specifications, like RFDiffusion; there are inverse protein folding models that go from structure to sequence, like ProteinMPNN. Then there’s fine-tuned models, like the many models that build on top of ESM to predict protein activity across diverse domains. All of this is enabled by dozens of open tools to browse proteins and sequences, find similar sequences, and visualize structure; and databases of protein structures, sequences, measurements and predictions that have been built over decades. Molecular dynamics and other conventional physical modeling tools like Rosetta have not gone away either, forming an important complementary set of tools to machine learning-based models. Physical models can also be used to train machine learning models for amortized inference.

Importantly, we have the ability to act on this information! We can synthesize sequences, put them in plasmids to duplicate them, pack them in AAVs, edit cells with CRISPR, verify our edits through sequencing, etc. Put together, this vastly accelerates the work of the synthetic or computational biologist. None of these models are perfect; but put together, they remove tedious and lengthy physical steps of optimization with surrogate endpoints, in silico. Surrogate endpoints are no stranger to biology: we use mouse models of disease all the time. But surrogates can be biased, and often fail the translation step to humans; the best surrogates are both fast and accurate.

Consider this recent Science paper on EVOLVEPro, a system to optimize protein designs. Protein design is a combinatorial problem. There are 20^N peptides with N residues, which for 100 residue proteins is larger than the number of atoms in the universe; exhaustively searching the space is a no-go. Rational design based on The results are outlined in the abstract:

EVOLVEpro outperformed zero-shot methods in benchmarks across 12 deep mutational scanning datasets, including epitope binding, nucleic acid binding, and enzyme catalysis. We used EVOLVEpro to engineer six different proteins with diverse applications. We improved the binding affinity two monoclonal antibodies by up to 40-fold, the indel formation activity of a miniature CRISPR nuclease by fivefold, the insertion efficiency of a prime editor by twofold, the integration efficiency of a serine integrase by fourfold, and the transcription fidelity and mRNA quality of a T7 RNA polymerase by 100-fold. (emphasis mine)

That sounds impressive! What is this mysterious system that can improve existing proteins on some metrics by 1-2 orders of magnitude? It’s conceptually simple: a foundation model for proteins is fitted with a prediction head, which is fit on baseline mutations to predict some desirable metric. The predicted best proteins are synthesized and assayed; the model is re-fit; the new best candidates are synthesized; and the process continues for a few iterations.

The model is built on top of ESM2, a foundation protein language model. ESM2 is a BERT-style masked language model: it takes a protein sequence and embeds each residue into a high-dimensional vector. It was trained on a large-scale dataset of protein sequences, UniProt. Once an embedding of a protein sequence is obtained, one can use it for different tasks, including prediction of structure (e.g. as in ESMFold); or one can average the embeddings to obtain a fixed-length digest, which can be used for the same kinds of things that you would use a sentence embedding language model for, e.g. RAG, recommendation, clustering, prediction, etc. On top of the average embedding, they put a prediction head, which is a simple random forest. They get initial data by synthesizing random variants; then, they engage in an active design process, using the model to predict the next mutation to try. The selection process is straightforward; they simulate all single residue changes, and pick the top N candidates that the model predicts.

If that sounds straightforward, it’s because it is: once you have a really strong base model, the ability to fit a basic regressor, and the ability to close the loop, you’re off to the races. Ultimately, all of this is made possible by the confluence of available tools, including the ability to read and write sequences at will; databases where people share their data; open models that can edited, re-mixed, fine-tuned and shared; and lab automation practices that make assays easier.

This translates into a step change in the speed of protein optimization, powered by a whole ecosystem of artifacts: foundation models, yes, but also datasets, databases, atlases, computational tools, and conventional models. As an aside, this new state of affairs has important applications in neuroscience, e.g. accelerating the design of new proteins for measuring and affecting the brain. So much of what neuroscience does is downstream of protein design: genetically encoded calcium and voltage indicators (GECIs, GEVIs); optogenetics; chemogenetics with DREADDs; barcoding, MAP-seq, BRIC-seq, Connectome-seq for mapping neural circuits. Heck, the biggest blockbuster brain drugs in decades are (modified) peptides: the GLP-1 agonists, which not only help regulate blood sugar and control body weight, but look promising to treat addiction and degenerative disorders like AD and PD.

Lessons to be learned

Let’s think about how these lessons from synbio would translate to the context of neuroscience. Are we there yet?

• Where’s the data? Does it have sufficiently high entropy to have a wide range of validity? Where’s our PDB and UniProt? Is the data sufficiently well-annotated and cross-referenced, in a format that facilitates learning at scale?

• What’s the coverage of our data? Do we have e.g. the equivalent of sequences from all the relevant organisms? A full genome from a single organism? Does it rise to the level of an atlas or a map, aspiring to have full coverage? A random sample? A non-random, non-representative sample?

• What means do we have to actuate the system? What are our knobs (e.g. the equivalent of the ability to synthesize new proteins)? Are our observations and models matched to our knobs?

• How are we validating our models? Do we have the means of taking our models and validating them in a closed loop, like we validate new protein designs?

On the data side, we have large-scale databases of neural activity like DANDI and OpenNeuro, as well as individual large-scale datasets like those from the Allen Institute, IBL, HCP, etc. The data can be substantial; around 10,000 hours or more from each of spikes, LFPs, sEEG, fMRI and EEG. This represents many PhDs worth of data collection, heroic efforts distributed across hundreds of laboratories. Clearly, these are a boon to the dry lab and the computational neuroscientist, and a starting point for foundation models.

But do they rise to the level of an atlas? It’s difficult to simultaneously obtain high physical coverage for recordings (i.e. covering the whole brain), spatial precision, and high coverage in task space. Even the highest dimensional recordings vastly undersample neural activity; if you add up all the spikes in the DANDI, that’s still–back of the envelope–less than the number of spikes you generate in your (human) brain every second. Furthermore, we’re getting, for the most part, relatively narrow slices of behavior, focusing on controlled and reproducible behaviors. Imagine building a DNA language model sampling from a single chromosome of a population of yeast cells; it’s not nothing, but it is a narrow slice of the space of possible genes, and a foundation model of the first yeast chromosome would have limited applicability outside of yeast genetics.

I’d argue that the more bio-centric atlases in neuroscience, e.g. cell type atlases, FlyWire, etc. tend to be more complete (i.e. have better coverage) than the neural activity atlases. But even in the best cases, we don’t always have the bridges that allow us to cross scales. FlyWire is wonderful as a complete connectome of the fly brain; but to translate that to simulated neural activity in a reliable way, we’d also want the transcriptomic background of each of the neurons, the distribution of receptors, and a complete characterization of the electrical activity of these neurons. This last one is an example of a bridge; the equivalent of PDB, something that takes us from one level (connectome → sequence) to another (electrical activity → protein folding). One of our highest priorities in the next 10 years of neuroscience should be to build the equivalent of PDB+UniProt for neuroscience in species that are phylogenetically close to humans; ideally, in humans themselves. That means genetic background; cell atlases and transcriptomes; mesoscale and microscale, molecularly-annotated connectomes; neural activity atlases; and calibration datasets (i.e. bridges) to stitch all these modalities into a coherent whole.

What about actuation? We’re certainly not at the level where we can synthesize neural activity the way we can synthesize proteins. The highest bandwidth and dimensionality means of actuation are at the sensory periphery (see, e.g. the retinal implants from Science, or even screens or headphones). Holographic optogenetics is still in its infancy, though I see a bright future for it. Focused ultrasound stimulation is rather low-dimensional, but it does have promise in precise modulation of neural activity in humans; and patterned stimulation across many sites in humans is still many years away. Finding new channels to precisely pattern stimulation is important; some of the recent work on biohybrid devices from Science and the next-gen neurotechnologies being funded by ARIA are moving us in the right direction.

Can we close the loop? I can think of a handful of cases where we have characterized, decoded, and optimized neural activity in a loop. One example, on the vision side, is that of inception loops, which have demonstrated finding maximizing stimuli for visual neurons (see refs 1, 2, 3, 4). A different example is nudging neural activity using holographic optogenetics (see refs 1, 2). These are currently heroic experiments, but they show a glimpse of what closed-loop design might look like. Foundation models, which are differentiable end-to-end, and can therefore be searched through gradient descent, have a clear role to play in closed-loop stimulation; but the means of stimulation and recording have to exist.

A call to action

All of that is a long-winded way of saying that our ambitions in neuroscience–solving all neurological disease, understanding intelligence and consciousness, etc.–are mismatched with our tools and datasets. Foundation models are one area of opportunity: leverage existing and future datasets to find good representations of neural data, make predictions, and optimize in a closed loop, leveraging the differentiability of deep learning models.

That cannot occur in a vacuum, however: the datasets that we collect, and the ecosystem of tools to read and write neural activity, cell identity, synapses, and connections should be matched in a virtuous circle. That will, almost assuredly, involve doing large-scale, hypothesis-free neuroscience focused on tooling and data, including in innovative structures like focused research organizations and coordinated research programs. Hypothesis-free science sometimes gets a bad reputation, but I think it can be most easily justified by epistemic humility: with billions of neurons, trillions of connections, thousands of cell types, hundreds of areas and receptors, maybe the first thing we should do is catalog these dang things, and then figure out how to poke at them to obtain causal models.

I can foresee, 10 years from now, many more neural atlases and databases, distilled into multiple application-specific foundation models and biophysically detailed models, cross-linked and validated through new neurotechnology. At the Amaranth Foundation, we philanthropically fund ambitious neuroscience projects led by ambitious mission-driven individuals; and my role is to lead our efforts in NeuroAI. Of course, I’m not a neutral party in all of this, but I think it’s a compelling vision for neuroscience.

But what does having a complete map of an animal brain give us? In this post, I outline how the FlyWire dataset was collected, what’s in the FlyWire codex, and what it can and can’t tell us about flies. As usual, we’ll get deep in the weeds, and then we’ll back to understand what it all means. I go through each of the papers of the collaboration to highlight how these artifacts are being used, and I’ll highlight some of my favorites from the perspective of systems and computational neuroscience. I ask questions about our ability, as scientists, to understand complex systems, our motivations, and what big neuroscience might look like in the near future.

How to build a map

FlyWire isn’t the first time a big chunk of fly cortex has been mapped: prior to this, the FlyEM hemi-brain connectome had been published in a collaboration between Janelia scientists and Google Brain. But FlyWire was the first effort to map the entire fly brain. Compare the size of the prior efforts to this one:

At a high level, the FlyWire collaboration constructed a map of the fly brain by:

1. Freezing a drosophila brain

2. Staining it with heavy metals

3. Slicing thin slices of the brain

4. Scanning those slices through serial electron microscopy

5. Aligning those slices

6. Using ANNs to (over-)segment the data into supervoxels

7. Using flood-filling ANNs to then join these supervoxels together into neuron reconstructions

8. Proofreading the resulting reconstructions by a manual process, reassigning supervoxels to the correct units to obtain clean reconstructions

The electron microscopy dataset has been around since 2018, published in Zheng et al. Reconstruction was no small feat: the dataset contains roughly 100 teravoxels. Tracing these dang neurons is a real challenge.

Segmentation and flood-filling are relatively straightforward applications of computer vision, albeit with the challenges associated with scaling. These methods do a first-pass reconstruction, but there are small errors. There are challenges ranging from missing slices to co-fasciculation and invaginations. There’s some non-zero probability of merging the current supervoxel with a wrong sibling–conversely, wrongly splitting two parts of the same neuron–and then the whole reconstruction is incorrect. The tyranny of long-range connectomics is that small error rates accumulate, and the probability of a clean reconstruction goes to 0 over a large volume.

There’s a very significant amount of work in proofreading–merging and splitting reconstructions until clean neurons are obtained. It took about 20 person-years to proofread this dataset, largely with the help of citizen scientists through the FlyWire platform. Here’s an example of a neuron before and after it was proofread (this is a mammalian basket cell, but they used the same CAVE software for proofreading the fly connectome).

That’s a lot of changes! Fly neurons have simpler morphology than that, but still, errors compound, and they must be tediously corrected. While 20 person-years sounds (and is) a long time, it’s only ~20 minutes for each of the 140k neurons in the fly brain. It adds up! Proofreading is a truly fascinating process of multi-step visual reasoning. I highly recommend watching this video tutorial to get a feel for the process.

Annotating the connectome

So we’re done, right? Au contraire! There are many annotations left to make. First, we need to detect synapses and determine their type. This was done in Eckstein et al. (2024), using existing known neuron types to come up with sparse annotations, classifying new synapses into one of eight potential neurotransmitter classes, then bubbling up that information to label each neuron’s neurotransmitter class, leveraging Dale’s law.

Then each neuron receives two sets of hierarchical annotations. On the one hand, neurons are classified by flow, superclass, class, and cell type.

There’s another set of annotations with respect to the developmental origin of the neurons, tracing different cell classes to lineage. The telltale sign that neurons that came from the same hemilineage is that they tend to aggregate in terms of their cell body positions as well as their tracts.

And so what you get at the end is:

• All the neurons, classified along two different hierarchies, their morphologies, and their neurotransmitter type.

• Their connection patterns at multiple levels of granularity. At a coarse level, the number of synapses between neurons (the weighted, signed directed adjacency matrix A, a sparse 140,000x140,000 matrix). At a granular level, the physical location of each synapse and the morphology of the connection.

What can you do with this? A guide to the FlyWire papers

With all of that under our belt, let’s go through the FlyWire papers listed in the Nature collection and see how they use this data. Note: I fed my sparse notes from reading these into Perplexity to fill in the gaps. I put two asteriaks ** next to papers that I think are of special interest to systems neuroscientists outside of the fly literature.

Studies involving just the FlyWire data and possibly other connectomes

**Neuronal wiring diagram of an adult brain

This foundational paper presents the complete connectome of an adult female Drosophila brain, comprising approximately 140,000 neurons and 54.5 million synapses. The authors analyze:

• Cross-hemispheric connections, revealing how information flows between brain halves

• Neuronal lineage assignments, providing insights into developmental origins

• A specific circuit related to ocelli, light-sensitive organs on the fly's head that aid in flight stability

The paper serves as a comprehensive "what's inside" guide to the fly brain, laying the groundwork for future detailed analyses.

Whole-brain annotation and multi-connectome cell typing of Drosophila

This study compares the FlyWire connectome with the previously published hemibrain connectome to assess neuronal stereotypy across individuals. Key findings include:

• Identification of 120 neuroblast lineages comprising 183 hemilineages, accounting for 88% of central brain neurons

• Analysis of neuron morphology, projections, and neurotransmitter identity within hemilineages

• Demonstration of overall stereotypy in cell types and strong connections, with some variations in weaker connections and specific brain regions (e.g., mushroom body)

This work provides crucial context for understanding the consistency and variability of neural circuits across individual flies.

Network statistics of the whole-brain connectome of Drosophila

Applying network science principles, this paper analyzes the connection patterns in the Drosophila brain. The authors examine:

• Global network properties such as degree distribution and clustering coefficients

• Recurring connectivity motifs and their prevalence in different brain regions

• The relationship between network structure and neurotransmitter identity

This analysis offers insights into the organizational principles of the fly brain from a systems perspective.

Parts list and connections for a visual system

Focusing on the fly's visual system, this study provides a comprehensive catalog of neuronal types and their connectivity patterns in the optic lobe. The authors:

• Identify and classify all intrinsic neurons in the optic lobe

• Analyze connectivity rules governing different cell types

• Explore how visual information is processed and relayed to higher brain centers

This detailed examination of the visual system serves as a model for understanding sensory processing in the fly brain.

**Predicting visual function by interpreting a neuronal wiring diagram

This study is dope! It uses connectome data to predict the receptive fields of various neuron subtypes in the visual system. Key aspects include:

• Development of a CNN-like model based on synaptic connectivity

• Prediction of neuronal function from structural data alone

• Identification of function for previously unrecorded neurons

This work demonstrates how connectome data can be leveraged to generate testable hypotheses about neural function.

The fly connectome reveals a path to the effectome

Using a coarse-grained representation of the connectome (signed, weight adjacency matrix), this study applies linear dynamics analysis to understand information flow in the fly brain. The authors:

• Derive eigenvalues and eigenvectors of neural activity assuming linear dynamics and short membrane time constants

• Identify dominant modes of neural activity and their relationship to behavior

• Propose a framework for linking connectome structure to behavioral outputs (the "effectome")

This theoretical approach provides a novel perspective on how brain structure relates to function at a systems level.

Diversity of visual inputs to Kenyon cells of the Drosophila mushroom body

This connectome-based study explores how visual information is encoded in the mushroom body, a brain region crucial for associative learning. The authors:

• Map visual input pathways to Kenyon cells, complementing existing knowledge of olfactory inputs

• Analyze the diversity and organization of visual inputs to different mushroom body compartments

• Compare visual and olfactory coding strategies in the mushroom body

This work fills a significant gap in our understanding of multimodal sensory integration in the fly brain, with implications for learning and memory processes.

Studies involving supplementary experiments

Some other studies that were part of this collection augmented the pure FlyWire data with extra experiments.

Heterogeneity of synaptic connectivity in the fly visual system

This study focuses on the variability of synaptic connections in the Drosophila visual system, particularly in Tm9 cells. Key points include:

• Documentation of higher connectivity variability in Tm9 cells compared to other cell types

• Utilization of expansion microscopy to verify findings across multiple individuals

• Implications for understanding the balance between stereotypy and variability in neural circuits

**A Drosophila computational brain model reveals sensorimotor processing

This paper is dope! It’s incredible that you can wire up a LIF model from the connectome data and it kind of works, even though, as we’ll see later, there’s a lot of missing information. It combines connectome-based simulations with experimental validation to explore sensorimotor processing in Drosophila. Highlights include:

• Development of a leaky integrate-and-fire (LIF) model based on the connectome

• Experimental verification of hypotheses generated by the computational model

• Insights into how sensory information is transformed into motor outputs in the fly brain

Studies involving primary experiments with a side role for connectomics

Other studies in this collection used the FlyWire data as one sub-part of an analysis; the role for connectomics was more subdued. Here the connectome is an enabler rather than a primary driver.

Hue selectivity from recurrent circuitry in Drosophila

This experimental study uses connectome data to support its findings on color processing in the fly visual system. Key aspects include:

• Primary focus on experimental investigation of hue selectivity

• Utilization of connectome data to create a model explaining recurrent connections' role in shaping hue selectivity

• Integration of functional and structural data to understand color processing mechanisms

Neural circuit mechanisms underlying context-specific halting in Drosophila

This research combines genetic screening with connectome analysis to investigate halting behavior in flies. Main points include:

• Genetic driver screening to identify circuits involved in halting behavior

• Confirmation of identified circuits using connectome data

• Demonstration of how connectomics can complement traditional neuroscience approaches

Studies around the same time not using the connectomics data, but relevant nonetheless

Let’s not forget all the very good studies that came out around the same time that didn’t use the FlyWire data, but used other connectomics data to do some very good systems neuro.

**Mapping model units to visual neurons reveals population code for social behaviour

While not directly using the FlyWire connectome, this study is relevant to understanding visual processing in Drosophila. Key points:

• Investigation of how visual information is encoded for social behavior

• Use of computational models to map visual neurons to behavioral outputs

• Insights into population coding in the fly visual system

This study is very cool–using virtualized environments to figure out what a fly must have seen at a particular point in time, and using that to analyze courting behavior.

**Connectome-constrained deep mechanistic networks predict neural responses across the fly visual system at single-neuron resolution

This preprint, though not part of the Nature collection, presents a relevant approach to integrating connectome data with functional predictions:

• Development of deep mechanistic networks constrained by connectome data

• Prediction of neural responses across the fly visual system at single-neuron resolution

• Demonstration of how structural data can inform functional models of neural circuits

This story is very cool because it solves an obvious problem with uncalibrated connectomics information: you still don’t completely know the synaptic strength nor the intrinsic properties of the neurons. Using task-driven neural networks can partially fill this gap.

What did we learn?

It’s truly remarkable how much mileage we can get out of a complete map of a nervous system. There is information to be gleaned from just the map itself; other times, to really make headway into understanding a problem, more data (e.g. electrophysiology, behavior, transcriptome) needs to be collected. Sometimes the map is the key artifact to create a model. Other times, the map just becomes background information; it would have been possible, but a real pain, to get at the relevant information. In all cases, there is something special about having complete information in one domain: it advances the boundary of science.

Is that it?

It’s important to recognize that even with all that information, there remains a lot of missing information that would be highly relevant for a computational model of the fly nervous system. For instance:

• The capacitance of the membrane (I learned in undergrad this was 1 uF/cm2, but apparently that can vary by a factor 2?! Wild.)

• The strength of each synapse

• The transfer function of each synapse

• The density of receptors at each synapse

• The density of ion channels as a function of the position across the membrane

That last one is particularly important to make good models of spike initiation. It’s recently been reported that spikes are initiated in fly neurons in a distal axon segment (DAS), an invertebrate analog of the axon initiation segment (AIS). There’s a high concentration of voltage-sensitive sodium channels (Para) in that segment, which is what allows spikes to start. But where’s the segment? It’s not as obvious as in vertebrates… it’s not next to the cell body!

So it’s clear that even with the morphology, the number of connections, their position, their “sign” (neurotransmitters), there are still many free parameters that would prevent us from emulating the fly brain 1:1.

Are we at the stamp collection stage of neuroscience?

For a long time, neuroscience has been driven in a hypothesis-driven way by a search for principles, motifs, patterns, and beauty. Optimality principles help us understand why things are the way they are, not just what they are or how they work. Yet, it must be the case that many features of the brain simply stem from evolutionary or developmental circumstances, which cannot be derived from first principles, so we must rely on direct measurement and mapping to understand them.

Thus, before we make pronouncements about why things are the way they are, we need to know what they are. We can work on this from two angles. In the first approach, we come up with some hypothesis about an unknown about a system, and we get data relevant to refuting or corroborating that hypothesis. A second approach is simply to document exhaustively what’s there in a richly annotated dataset (a “map”), which covers systems and hypotheses that we might not even have thought of beforehand. This is a lot more initial work, but then follow-up work is vastly accelerated.

The recent FlyWire work underscores the value of having a complete characterization of a system. While these efforts at large-scale mapping can sometimes be dismissed as mere “stamp collecting,” these projects are in fact crucial for laying the groundwork for hypothesis-driven research. Having a complete connectome can illuminate vital details that guide more targeted studies.

More connectomes or more maps?

Of course, connectomes do not address every question—especially in mammalian species where variability and plasticity are the norm. The success of FlyWire shows the promise of connectomics, but for different organisms and use cases, different types of maps are going to be relevant: spatial transcriptomic, functional maps, cell atlases can partially elucidate neural circuits in different organisms. It is up to us to make judgment calls about the most valuable maps, but my point is that maps are good.

One obvious challenge with this kind of bottom-up science is data volume. In Drosophila, the full network involves around 140,000 neurons. The many recent FlyWire publications capture only a fraction of what is there. Relying on manual, bottom-up approaches to digest this complexity becomes increasingly impractical. And that can feel agency-depleting for those of us who believe in the value of small, slow science: all trees, no forest.

Deciding what to study and characterize involves a mix of intuition, aesthetics, and pragmatism. We often focus on circuits like the compass system because they are aesthetically pleasing, relatively tractable and well-characterized, yet still hold unanswered questions. As data grows, human effort alone–grad students squinting at plot trying to see the forest through the trees–might not be enough. Freeing ourselves from our cognitive bottlenecks—say, our inability to hold information in our heads about the intricate connection patterns of hundreds of thousands of neurons—will likely become essential in detecting patterns and suggesting new directions for investigation. That’s the promise of AI agents, crawling the scientific literature and maps for new insights to be tested: renewing our commitment to hypothesis-driven science, supported by extensive mapping.

Ultimately, Drosophila studies highlight both the power of maps. By anchoring our thinking in robust, empirical data, we gain a sharper view of how evolution, development, and function intertwine to shape neural circuits.

There’s a lot of negative discourse around ML tools, including from scientists (some examples). It’s easy to use these tools in agency-depleting ways: doing superficial readings of texts and summarization rather than deeply engaging in papers; have code LLMs write bad code that cause a giant pile of tech debt; and delegating the fun, rewarding parts of your work to an unthinking machine. But they can also be used in agency-enhancing ways. My goal is to give you a path toward the latter.

Meta: choosing the right ML tool

Someone on X1 mentioned that they couldn’t figure out how to spend more 100$ a month on ML tools2, and this was worrisome. This is perhaps a strange framing, but here’s the reasoning: if you’re a highly-paid knowledge worker, if you could save hours per month by using AI tools, you should be jumping on every occasion to use them, and price them consequently. Here we are with the Library of Alexandria at our fingertips–albeit a stochastic one–for 20$ a month, and yet a lot of us will periodically cancel our subscriptions because we don’t actually use them. What’s going on?

The main ML tools have a huge discovery problem: it’s really hard to figure out what they can and can’t do. Using the tools is also a very private thing, so while your co-worker might have figured out a valuable workflow, you wouldn’t ever encounter it yourself unless you actively search for it. Workshops where people share their use cases and pain points are a great way to get up to speed.

In the absence of that, there are some great resources to learn about how to use AI tools and stay up-to-date:

• Ethan Mollick’s blog. Ethan is a professor at the Wharton School at UPenn who has become something of an AI tools influencer. His bi-yearly summary of which tools bring the most value is always a great read.

• Nicholas Carlini, How I use “AI”. A great, long blog post on different use cases for AI tools from the perspective of a software engineer. He shows the actual prompts he uses.

• AI News. A daily AI-generated newsletter that tracks developments in AI tools, mostly LLMs. This one is highly technical, probably too in the weeds for most, but it’s a great way to catch up on the SOTA without digging into every last Reddit and Discord post.

Gotchas: there is a new tool that comes out every day. You can easily waste a lot of time testing out new tools that add marginal value. There are also marginal value AI influencers all over X and LinkedIn. It’s ok to lag behind in adoption by months.

Chat

There were many professors in the audience, so it comes as no surprise that chat tools that generate text were popular. The use cases are endless:

• Summarize long documents

• Help craft derived text, say, a tweeprint or a status report

• Expand a list of bullet points into a proper email

• Help battle the dragon that is Overleaf and its overfull hboxes

• Catch grammatical mistakes and unclear sentences (especially relevant to people for whom English is a second or third language)

• Give a high-level overview of a field, from which you can branch out

• Get over the line when you’re just finishing a doc and need a fresh set of eyes on it; conversely, get you out of your head when you’re trying to write a first draft

One attendee mentioned an interesting mental model for chat models: they’re very good at style transfer. Taking text in one format and putting it into another format is really their forte. Now personally, I don’t always like the text that it generates in its house style, as it tends to be lifeless. I have the same qualms about Grammarly: sometimes, I do want to place redundant emphasis, create run-on sentences, and generally do all the things my high-school English teacher warned me not to do. As a rule, I don’t use chat tools (much) for blogging, because I want my blog posts to sound like me.

Chat tools go far beyond the humdrum use cases of automating the boring stuff, although their abilities can vary by tool. They don’t just do text-based tasks: paste in a screenshot or a PDF and it will generally do a good job of ingesting the content.

Here’s my rollup of useful general-purpose LLM tools:

• Claude. My personal favorite, as I tend to prefer its style over ChatGPT’s–obviously, this is highly subjective. Excels at writing code. Again, worth the 20$/mo. This is my daily driver but I also pay for ChatGPT.

• ChatGPT. A general-purpose tool for everyday use. Several people mentioned that o1-preview in particular excels at mathematical reasoning.

• NotebookLM. Most known for its flashy podcast generation feature, an impressive gimmick whose sheen rapidly wears off. The real standout is its giant context window (2M tokens) and ability to cite references. You can throw in dozens of PDFs or several books worth, and it can do things like cross-referencing. Looking for a long PDF to try it on? How about our recent 92-page roadmap on NeuroAI safety?

Gotchas: many tools allow the owner of the tool can train on you data. You should think twice about sharing private information with these tools, whether that’s unreviewed papers, private IP, or PII.

Aside: Audio as an interface to LLMs

Speech-to-text is a natural complement to LLMs. One very pragmatic use case is taking meeting notes. If you’re talking to a lot of people (we’ve talked with 100+ neuroscientists as part of building up the Enigma Project and then our NeuroAI safety roadmap), you’re going to take a lot of notes, which is boring but crucial work. I add Fireflies to all my meetings instead, and then interrogate the transcripts using an LLM.

I also use audio-to-text as a tool for thought. Sometimes I’ll read a paper and then record a conversation where I work out what I’ve understood from that paper. Or I’ll dictate an often rambling review for a conference paper and then ask an LLM to reformat it into a readable format. Audio untethers you from your laptop for deep thinking, allowing you to go on long walks and think through a problem.

Code

[Slide deck for my talk]

I led a section on coding tools. 80% of software engineers now use LLM tools, mostly Github Copilot, and Copilot is now a billion-dollar business. It is likely that PhD students and postdocs in your lab use them as well.

I use code tools all the time, mostly Copilot inside of VSCode, Claude, and occasionally Cursor. They have saved my butt more than a few times, with some prominent examples being:

• I spent 3 weeks reviewing and editing all the code and text for all the tutorials for the NeuroAI course for Neuromatch. It was an absolute grind, and pretty high stakes given our hundreds of students. I would have not been able to make that much progress without Copilot inside of VSCode’s jupyter notebook environment.

• I created a viral website ismy.blue to test your blue/green boundary and teach the general public about psychophysics of color perception. It received 2M visits and was covered in The Guardian and Wall Street Journal. I am not very proficient in HTML/JS, so this extended my reach.

• I made a website with my co-author Joanne for our NeuroAI safety roadmap. It involved using pandoc to generate HTML from LaTeX, creating a dev server with live reload to test out the output, and doing a ton of CSS wrangling. Doing this manually would have been a huge pain.

• While working on the NeuroAI safety roadmap, I made a strategic mistake: I drafted everything in Google Docs. Once I realized the paper was 3X longer than I originally intended and gdocs was slowing down to a crawl, I exported from gdocs to Word and then to LaTeX. Unfortunately, there was a lot of extra markup generated and the doc was slightly broken. Claude was fantastic at correcting the LaTeX markup.

• I translated this app for a colorimeter from Matlab–for which I don’t have a license–to Python via Claude. This took a few minutes, whereas a manual effort would have taken a couple of days.

I showcased several use cases, including creating interactive visualizations from Python code, translating straight Python to Cython to obtain a massive 200X speed improvement in a leaky-integrate-and-fire simulation, and generating a skeleton for an adversarially robust MNIST classifier. Affordances for working with code are quite advanced compared to working with text:

• you can use Copilot for autocomplete

• you can interrogate an LLM in natural language from within your coding interface or in a sidebar

• Claude’s artifacts and ChatGPT’s canvases give a specialized interface to write code and optionally preview it

• Cursor can generate diffs and integrate them in your files

Code tools are very powerful, but they also create an opportunity to massively shoot yourself in the foot. The usual way you generate code in science is writing a little bit, testing it out, checking it looks good, and moving on. The tools generate too much code too fast for this to be an effective strategy, and can easily create tech debt traps.

For example, ismy.blue used a snippet of code to fit a psychometric curve (logistic regression). Because there’s no sci-kit equivalent in Javascript, I asked Claude to write it from scratch using Newton’s method. I’ve written this loop many times before in Matlab, and when I looked at it, it passed the sniff test, so I went ahead and copy-pasted it into the website code. Big mistake! It turns it didn’t check for convergence and about 5% of visitors to the site got the wrong boundary. The solution was a backtracking line search that should have been there from the get-go. If I had written the code from scratch, of course I would have written a backtracking line search. Argh!

The solution is defensive programming: write tests that check critical code sections for correctness. I have a short section on this idea on goodresearch.dev, which I hope to expand in the future.

Aside: writing code vs. mentoring

By default, LLMs will try to solve your problem directly. This can lead to become overly reliant on LLM code tools rather than developing proficiency. You can, however, prompt LLMs to give you mentoring and guidance rather than writing your code.

For instance, I’m learning Rust as part of Advent of Code, a holiday-themed daily coding challenge. On day 4, Advent Of Code’s question relates to solving a word search problem. I can ask Claude to directly solve the problem, and it will oblige:

Then I can ask questions about its solution. In this case, it created a unit test using tags, a syntax I’m not familiar with. So I can dig in and find out why it did the things it did.

That’s certainly better than copying and pasting the code without thinking about it. However, I can go further by instead asking Claude to be a socratic tutor to help me solve the problem. Here’s one prompt that does the trick:

I am familiar with Python and JS and am attempting to learn Rust. I was a SWE at a FAANG and am familiar with data structures and algorithms. To learn Rust, I am doing daily exercises through the Advent of Code, a daily holiday-themed coding challenge. Help me guide my learning by asking questions in the style of a socratic tutor. Here’s today’s question: <question>

Instead of giving me the answer to the question, it prompts me to reflect:

I can then have a conversation with Claude and activately engage in the learning:

I can also give it broken implementations of code and help me better understand Rust’s syntax, or ask qualifying questions. This approach of “don’t give me the answer, give me the tools to answer the question myself” is useful beyond code: you can use that to decide which tack to take for a paper, a math proof, planning a conference, preparing for a meeting, etc.

Papers

Brad Love presented his paper on predicting neuroscience results using an LLM, an intriguing future use case that I’m excited for3. In the here and now, LLMs are already powerful enough for searching through papers and extracting structured information from them.

Probably one of the most useful everyday tools is Perplexity Pro. It’s an LLM-powered search engine. In Pro mode, you can set it so it searches from scholarly archives (i.e. Nature, arXiv, biorXiv, etc.). It only gives you about 8 or so references per query, but its hit rate is high, and it feels much better for freeform search than Google Scholar. I was skeptical of this tool, but it got high notes from Sam Rodriques from FutureHouse, which is building AI tools for scientists.

There is a slew of other tools for more specialized use cases. Elicit is a tool to facilitate structured searches, including metaanalyses. It can extract structured information on demand. Has Anyone is a tool from FutureHouse to determine whether something has ever been done. PaperQA2 is a FutureHouse agent-based model that can be used to do deep searches through literature.

It’s also possible to do searches using the raw models themselves for power users. I used this in our recent roadmap to estimate the rate of doubling in electrophysiology and calcium imaging capacity. I used the API version of GPT-4o to filter and parse through all 40,000 bioRxiv neuroscience abstracts, then ran the tool again to extract number of neurons, probes, model system, etc. This is monk’s work that would have taken many weeks of research assistant time, but that I was able to complete in about 3 days.

Unlike the case for code, it’s fair to say that the affordances for these tools still have a lot of gaps. For instance, I’d like to be able to search my paperpile library for relevant papers and dump it in NotebookLM. Simple enough, but right now I have to manually select relevant papers and download every single PDF before uploading to NotebookLM. Amelia Wattenberger showcased a tool to do search and download for arXiv for precisely this use case, which showed that UX tweaks could really make a large difference here. There is a lot more to be done in this space, and it seems clear that FutureHouse in particular is focused on making many of the existing tools more precise and usable.

Graphics

I feel like this may be the largest gap in the arsenal of ML tools for scientists. Dall-E’s house style sticks out like a sore thumb. Yes, you can get it to create good illustrations, but it still leaves a lot to be desired in terms of composition, aesthetics and accuracy. My gotos for generating illustrations for papers remains a combination of free icon libraries, stock photos, BioRender, Inkscape, LucidChart, Canva, tikz, etc.

The biggest issue is that I want to have fine control over aesthetics. Generated text is easily editable post-hoc. Images–not so much. What I really want is a vector image generation tool. This is slowly coming together. I tested out Illustrator’s new AI tools, and was able to generate a vector monkey with VR goggles that was editable and was not half bad.

Conclusion

It can be exhausting to keep track of all the latest advances in AI and how they impact your daily productivity. Nevertheless, adoption has been rapid. Code AI tools represent one of the most mature categories with some of the best affordances–Copilot was released before ChatGPT. Text tools are daily drivers that suffer from a lack of discoverability, which can be alleviated by sharing workflows with colleagues and reading blogs.

Tools to read and absorb papers are still in their infancy, but they could be transformative: WikiCrow is an automated system that generated 15,000 high-quality Wikipedia-style articles on human genes that lack one. Bad quality AI art is all too common, but an AI-first vector editing tool would be highly compelling. All in all, we have the ability to do things that sounded like science-fiction just a few years ago, but we need to be judicious in how we deploy these tools.

Linear regression is a well-accepted way of comparing artificial neural networks and brain activity (I wrote a tutorial-like intro here). Here’s how that usually goes:

• You collect lots of responses from the brain in response to stimuli (images, sounds, etc.)

• You do the same for an ANN

• You regress the ANN onto the brain, keeping the weights of the ANN fixed

• How well that regression works in predicting data in a validation fold tells you something how good a model of the brain is that ANN

Or does it? The position paper Maximizing Neural Regression Scores May Not Identify Good Models of the Brain argues that, well, it’s not that simple. There was a slew of activity on X1 about the paper: some supportive, some negative, some sympathetic but saying that the paper went too far in the skeptical direction and overstated their argument. I will try to synthesize the discussion here and give some context so you can navigate this field.

My TL;DR: Neural scores don’t always mean what we think they mean. When neural scores are calculated between brains and the highly overparametrized representations of neural networks for a small number of stimuli, we have to carefully interpret the scores.

Breaking down the argument in Schaeffer et al. (2024)

Schaeffer and co. build an argument that neural regression score maximization doesn’t tell you which model is more like the brain. The argument goes like this:

1. Dimensionality is highly predictive of neural scores. Big networks with many features that cover a lot of ground generally match the brain better than small networks with low dimensionality. This conjecture was first made a few years ago to explain some puzzling results in entorhinal cortex and hippocampus (Schaeffer et al. 2022; although see Aran Nayebi’s rejoinder on X), and others found congruent results (e.g. Elmoznino and Bonner, 2022).

2. Neural scores are a reflection of the inductive biases of linear regression.

The effective dimensionality argument (1)–bigger is better–is intuitively attractive, but it’s wrong. While there’s some correlation between dimensionality and neural scores, the exact relationship is explained by the more subtle spectral theory of regression by Canatar et al. (2023), an excellent paper. It’s not just effective dimensionality (=big network good) that matters; it’s also the projections of the eigenvectors of the design matrix on the observations (=bigger not always better). To put it in the words of Mick Bonner: “High effective dimensionality alone is not sufficient to yield strong performance”.

Neural scores in the interpolation regime

What about the second point on the inductive biases of regression? I wish the authors had focused more on this, because it is an important and subtle point. When you really, deeply think about it, it is strange that we:

1. probe brains and models with hundreds of stimuli

2. map models to brains using linear regressions with thousands, sometimes millions of parameters

3. use those to differentiate between different models

The classic statistical viewpoint says that in this overparametrized regime, there is no unique solution to the linear regression. Equivalently, an infinite number of regressions could fit the data equally well, and you need assumptions to solve this degeneracy. The assumptions must be matched to the problem at hand and they influence the outcome of statistical comparisons.

Now, Schaeffer et al. don’t spell out why the inductive biases of linear regression might be problematic. It’s worth going back to spectral theory to understand why. Canatar et al. study the highly overparametrized regime (P stimuli << M units), where there are far more units (i.e. neurons, design matrix columns) than there are stimuli (i.e. observations, rows of the design matrix). Their theory is valid when there is no observation noise and the mapping between stimuli and neurons is static. They define the generalization error by chopping up the full design matrix into train and test. The generalization error is the result of testing on the test set with the linear regression weights inferred on the train set. With their definition, the generalization error must go be exactly zero as the proportion of training examples tends to 100% (remember, no noise!). You can see this, for example, in their figure SI.4.1.

In this setup, all models eventually get to 0 error. The path that models take to that zero error is a bit unintuitive, and it doesn’t map neatly into the usual schema of low error = good model that aligns well to the brain, high error = bad model that is unlike the brain. It depends on the details of the spectrum of the model activations matrix. For example, in the plot above the right, layer 3 and layer 4 swap places depending on the proportion of the data in the train and test set. That nonmonoticity is the product of the inductive biases of linear regression, not so much whether a model is brain-like or not.

TL;DR: take your scores with a grain of salt and use multiple lines of evidence to show that an ANN is like the brain. In the highly overparametrized regime, neural scores can have an unintuitive and complex relationship to the thing we ultimately care about: how well-matched an ANN is to the brain.

Contra Goodhart’s law

What happens when we overinterpret and hyperfocus on a single fixed metric? One take on X is that “as the field optimizes against a fixed proxy metric [which doesn’t fully capture brain similarity], the field begins Goodharting”. Goodhart’s law states that a metric that becomes an objective ceases to be a good metric.

So, are we hacking our metrics? I don’t think so. One pattern that’s associated with Goodhart’s law is that benchmarks get quickly saturated as people teach to the test, as is the case with LLMs. If we were Goodharting, the metrics would go only one way: up. But if you look at the history of vision BrainScore in particular, the metrics have stayed relatively stable over the years, perhaps even decreasing.

To be clear, this is not a good thing either! We want our theories to rise in accuracy over time so that we get ever closer to the truth! This is a point that I investigate in detail in the roadmap on neuroscience for AI safety that we recently published. Ideally, we’d be able to measure a low-noise gradient in model space to uncover more brain-like models over time. So far, we haven’t found great ways of doing that, which is a core problem for the field. We propose a few different lines of research that could help leapfrog this problem. In any case, I don’t think that it’s true that we are falling pray to Goodhart’s law.

Maybe many metrics aren’t good either

The singular focus on the flaws of one particular metric, in one regime, can make us forget about how neural scores are often used in practice. Neural regression scores are often tested out-of-distribution, which the in-distribution setup of Canatar et al. doesn’t cover. And we often use more metrics than just linear regression. Using more metrics–RSA, DSA, CKA, looking at tuning curves, one-to-one-mappings–is a commonly suggested way of avoiding becoming overly reliant on a single top-line metric. This has now become a best practice in the field: many models, many datasets, and ideally many metrics, each testing the model along different axes of variance. Although many papers still focus on one model and metric to their detriment, there are plenty of examples of papers that build a more compelling story around comprehensive testing, including Zhuang et al. (2021), Mineault et al. (2021), and Conwell et al. (2023).

Except… sometimes the metrics don’t agree and the conclusions are quite sensitive to the choice of metric. This was the conclusion in Soni et al. (2024) and Klabunde et al. (2024). That’s worrisome.

Metrics can vary a lot in how they rank different models. No metric does universally better than any other

My best guess for why this is boils down to two things:

1. Bad metrics: Some of the metrics are genuinely not good, or not used in practice, which adds noise to the rankings and inflates the extent of the problem.

2. P << M: there are many fewer stimuli than the dimensionality of the activations. That means the comparisons rely on inductive biases which are introduced in subtle and unintuitive ways. With a small number of stimuli, you don’t get good coverage of the distribution of inputs either.

The first point is solvable; Alex Williams and his group, among others, have been very active in finding axiomatically good metrics (e.g. this work on shape metrics). But the P << M problem remains.

What else can we do?

If the true underlying problem is overparametrization and the subtle inductive biases that are introduced to avoid the issues that come with overparametrization, maybe we ought to do things differently. What are some potential solutions?

1. Increase P. We should measure the responses to more and higher entropy stimuli.

2. Target stimuli better. Instead of wasting a big chunk of the P on constraining dimensions of the models we don’t care about, use targeted stimuli designed to maximally differentiate models. This is the solution proposed by Golan et al. (2020).

3. Decrease M. Use more constrained functions to map from ANN to brains.

Many people have argued for 1 and 2, and they’re pretty uncontroversial. Here I’ll try to make an argument in favor of 3, which I haven’t seen before. With 3–using a highly constrained set of weights–we’re also introducing an inductive bias, but we’re making it explicit. Even in this day and age of large-scale models with millions of parameters, P >> M is a viable regime. For example, the models explored by Lurz et al. (2021) have M < 100, but P up to 16,000. All it takes is a highly contained receptive field model to get into the classic regime.

It might seem like we’re skipping over the beautiful theory of double descent, but in fact the overparametrized regime (the double descent regime) doesn’t always lead to better solutions when there’s noise in the observations (see this tutorial for example). While the theory of Canatar et al. (2023) has a large surface, it assumes no noise in the measured responses. We have a theory of scaling laws for linear regression in the underparametrized, noisy setting in the NeuroAI safety roadmap. I’d love to see an extension that fuses the theory of Canatar and ours.

Parting words

A model that explains more of the variance is better than a model that explains less of the variance, all else being equal. Maximizing explained variance is the way to go if we want to learn digital twins of the sensory cortex and control population activity. But a model that accounts for more variance in the brain is not necessarily a better model of the brain. And it’s easy to read more into a single score than what is warranted.

Antonello and Huth (2024) report that a next-token prediction objective and a German-to-English translation objective lead to representations that are equally good at predicting brain responses to podcasts in unilingual English speakers; the authors argue that “the high performance of these [next token prediction] models should not be construed as positive evidence in support of a theory of predictive coding [...] the prediction task which these [models] attempt to solve is simply one way out of many to discover useful linguistic features”.

Structurally, I think that what we want is for our scores to point us the way: to tell us where to look for more brain-like models on the mountain which is model space. But the scores are noisy and our hill-climbing is treacherous. Until we can get to p → ♾️, we will have to carefully consider the value of our models and scores.

Acknowledgements: Thanks to Shahab Bakhtiari, Aran Nayebi and Tim Kietzmann for reviewing this piece. Aran has an insightful comment on openreview breaking down the paper’s claims in detail.

I think you’ll like it. There’s lots of technical analysis on NeuroAI: data acquisition capabilities and available data across a wide range of relevant modalities, from electrophysiology to connectomics. We cover 7 different paths where NeuroAI can impact AI safety. It’s also a bit of a time capsule and a love letter to NeuroAI; while it’s by no means a comprehensive review of the whole field, it’s about as comprehensive as it could be without becoming an entire book.

I’d love to hear your thoughts. We made a companion website to make it easier to read on the go. You’ll have to pace yourself with this one as it clocks in at 22,000 words, but it can be read mostly out-of-order. I can almost guarantee you will learn something.

• Website: neuroaisafety.com

• Paper on arXiv

Here’s a little flavor and a walkthrough of the roadmap to orient yourself.

NeuroAI, fast and slow

NeuroAI is a field that takes inspiration from AI to help us understand the brain, and vice-versa. The neuro→AI route has been focused thus far on bringing new capabilities to AI, inspired by neuroscience and psychology: robustness to adversarial and out-of-distribution stimuli, higher data efficiency, smart and complex neurons, an active learning phase inspired by development, etc.

Naysayers will point out AI has been racing in capability without much neuroscience input. It is hard to find a benchmark that AI has not saturated, with the ARC challenge being a notable exception. The reason AI has advanced without being anchored in neuroscience is no secret: neuroscience is slow. You can get a lot more reps in with purely in silico experimentation, unanchored by slow wet lab experiments. Thus, the canonical examples of neuroscience influencing AI are decades old: the perceptron, ANNs, CNNs, and RL. Where does that leave the promise of advancing AI through neuroscience?

Solving a real problem: safety

AI capabilities are increasing, but safety remains to be solved. We’re at the alchemy stage of AI: we have some empirical findings, but we don’t have a general science of intelligence, or how to control it. What if NeuroAI focused on improving AI safety?

To give you one practical example of an unsolved problem in AI safety, consider adversarial examples. Take this photo of my dog Marvin, which is correctly classified as a chihuahua by a pretrained model, and add a little bit of imperceptible, targeted noise to it. Now it’s confidently classified as a microwave1.

There have been over 10,000 articles on adversarial examples, and we have yet to solve them. A recent study on scaling adversarial training showed that you would need multiples of GPT-4 compute to solve adversarial robustness on a toy task, CIFAR-10. It’s a giant security hole that will only get amplified as we get multimodal, semi-autonomous agents that can act in the world.

A good way of thinking about this problem, pioneered by Ilyas et al. (2019), is in terms of robust and non-robust features. Many features can lead to correct classification. Only some are robust out-of-distribution.

The more general point is that there are many ways of getting to intelligent behavior, but most of these are not human-like or safe. Recapitulating behavior through imitation learning works in-distribution, but it's an underconstrained problem out-of-distribution.

Better AI through constraints

The human brain has a number of mechanisms for flexible and safer intelligence. We've evolved sophisticated mechanisms for safe exploration, graceful handling of novel situations, and cooperation. Understanding and reverse-engineering these neural mechanisms could be key to developing AI systems that are aligned with human values.

The human brain might seem like a counterintuitive model for developing safe AI systems: we wage war, we’re biased, and often fall short of our lofty ambitions. But we don’t have to import brains wholesale: we can focus on emulating behaviors and computations that are useful from an AI safety perspective. We call this a selective approach toward studying the brain for AI safety.

The general premise is that more constraints from human brain biophysics, representations, and behavior have a higher probability of leading to the basin of safe, human-like solutions. This is a point that was made previously by Andreas Tolias, one of the co-authors of our roadmap, in Sinz et al. (2019). They framed it in terms of strong generalization, but you can make the same point about safety more generally.

If you agree with that premise, then the path toward AI safety through neuroscience is to find sources of constraints. That could be done at multiple Marr’s levels, and which approach is most promising is both 1) an empirical question about the effectiveness of each constraint and 2) a question of which approach will get you there fastest given technological bottlenecks in data acquisition and analysis. We tackle both questions in this roadmap.

Why NeuroAI safety now?

There are a few reasons that have prevented NeuroAI from tackling AI safety. The first is that the AI safety literature is hermetic and neuroscientists, by and large, have not deeply engaged in it. It’s speculative, it moves fast, and it’s hard to get a lay of the land. One of our goals with the roadmap was to make the AI safety literature approachable to neuroscientists by introducing a common framework that we could all refer to. We adapted a framework from Deepmind (2017) for different ways in which AI can be made safer, and refer back to it throughout.

A second reason is that while there have been a few proposals for how exactly NeuroAI could impact AI safety, these have mostly been at a high level. I haven’t seen a lot of deep technical discussion and analysis on how to get there, and this roadmap fills the gap.

A third reason is tooling. We know a lot more about the brain than we did a decade ago, most notably in the fly. Still, our understanding is partial. If neuroscience is to meaningfully contribute to AI safety, we need to dramatically accelerate our ability to record, analyze, simulate, and understand neural systems. The catalysts for large-scale neuroscience are already here, thanks in part to massive investments made by the BRAIN Initiative in the past decade. We should take advantage of this moment to learn more about the brain and potentially use that knowledge to impact AI safety. Even if the impact on AI safety is smaller than expected, we’ll still make progress in understanding the most mysterious object in the universe, advancing neurotechnology along the way.

Highlights from the proposals

We've organized our roadmap around 7 themes. For each, we perform in-depth technical analysis, identify key bottlenecks and make recommendations for further research and investment:

• Reverse-engineer the representations of sensory systems. Understanding how the brain achieves robust perception and handles novel situations could help us build AI systems that are more resistant to adversarial attacks and better at generalizing to new situations. We derived scaling laws for sensory digital twins to determine how much data we’d need to build digital twins of single sensory neurons–they have log-sigmoid scaling curves, which I haven’t seen documented before. We then established one proof-of-concept for how the resulting digital twins could help make AI systems more robust. There is a lot more to be done here.

• Create embodied digital twins. Functional simulations of brain activity combined with physical models of bodies and environments could help us understand how embodied cognition contributes to safe and robust behavior. I discuss ongoing work in building virtual animal models and foundation models that could help us get there.

• Develop detailed simulations. Creating detailed biophysical simulations of neural circuits could capture the fundamental constraints of biological intelligence, which serve as templates for building safer AI systems. Connectomics has had a quiet revolution over the last couple of years, and the costs have fallen precipitously from last year’s estimate of 15B$ for a whole-mouse connectome. E11 Bio, lead by our co-author Andrew Payne, projects that they can achieve 100-fold reduction in cost in 5 years. We have a lot of great background material on light-sheet expansion microscopy and barcoding and discussions on compute necessary to simulate an entire nervous system.

• Build better cognitive architectures. Based on our understanding of how the brain implements capabilities like theory of mind, causal reasoning, and cooperation, we could build modular, probabilistic and transparent cognitive architectures that better align with human values and intentions. This section was contributed by Basis and their collaborators, experts in cognitive architectures. Probabilistic models of cognition are having a moment, and it might be the right time to build an analog of PyTorch for Bayesian cognitive architectures, together with large-scale naturalistic datasets.

• Advance brain-informed process supervision. Using neural and behavioral data, we could fine-tune existing AI models to better align with brains and encourage safe behavior. This is a surprisingly under-studied area, accessible to many given the existence of large-scale open fMRI datasets. I had to rewrite this section in October in light of this recent paper from Mariya Toneva’s lab showing a proof-of-concept in fine-tuning audio models with brain data; things move quickly!

• Reverse-engineer loss functions of the brain. Using functional, structural and behavioral data to determine the loss functions of the brain, we could derive better training objectives for AI systems. We tried to tackle a mystery that has bugged me for a little while: why have visual brain scores stagnated? Can the methods of task-driven neural networks uncover the brain’s true loss functions? And can we use the mapping between RL and the brain to uncover the brain’s reward function? This could be the basis for a robust research program.

• Leverage neuroscience-inspired methods for mechanistic interpretability. At a meta-level, we could apply the tools of neuroscience we use to study biological neural networks to understand artificial ones, and vice-versa. This could help make AI systems more transparent and verifiable, and help us accelerate learning from the brain. I’m impressed with the advances in mechanistic interpretability over the last couple of years, which I hope we port over to neuroscience soon; more on that in a later blog post.

Video: Optical reconstruction of brain circuits, from E11Bio.

Now, the paper can be read out of order, but the approaches are not independent – progress in one area means progress in others. What we need to advance AI safety through neuroscience is a coordinated effort that advances them all in parallel. This means investing in neurotechnology development, scaling up neural recording capabilities, and building neural models at scale across abstraction levels.

Once again, the whole roadmap can be read on:

• Website: neuroaisafety.com

• Paper on arXiv

Please reach out to neuroaisafety@amaranth.foundation for comments and next steps.

This post will start less technical than usual, but it will ramp up in later sections. I would love to hear your thoughts in the comments, via email or on X. Let’s go!

What’s a foundation model, anyway?

A foundation model is an AI model that serves as a foundation–a basis–for multiple downstream use cases. Some defining characteristics of a foundation model include:

• It’s pre-trained in a self-supervised or unsupervised way to learn the distribution of the data it attempts to model

• It is then finetuned to allow it to perform a downstream task

• It is trained on large-scale data

• It leverages large-scale compute

• It contains a large number of parameters

• It leverages generic architectures with weak inductive biases–mostly transformers, but also convolutional neural networks and state-space models–that are well adapted to training at scale in highly performant clusters

• Its performance on its pre-training task is predictable as a function of data size, compute, and model size–scaling laws

• They may display emergent properties not immediately visible in the pre-training task

  • Large language models, in particular, display emergent properties such as reasoning and in-context learning, despite being pre-trained on next-token prediction

• A foundation model is composable; its trained weights, the result of large-scale training, can be used to initialize larger systems, which may include other foundation models or conventional machine learning models as subcomponents.

This setup in terms of stages of pre-training and several stages of fine-tuning was popularized by ULMFit, which proposed this pipeline for natural language processing:

Foundation models include:

• Large language models, LLMs, such as GPT-4, Gemini, Claude or Llama

• Vision models such as ViT-22B or Segment Anything Model (SAM-2)

• Generative vision models such as Dall-E, Stable Diffusion and MidJourney

• Speech-to-text models such as Whisper or DeepSpeech

• Generative audio models such as Udio or Suno

• Vision-language models (VLMs) such as CLIP and SLIP

• Vision-action models such as PaLM-SayCan

• DNA, RNA and protein foundation models such as Evo, AlphaFold, RoseTTAFold and ESM-3

Models have different levels of openness, including:

• Closed source, usable via a web of chat interface

  • Perhaps the model is available via an API, which facilitates automation

• Open weights, which means that the model and its checkpoints are available, but training scripts and data might not be available

• Open source, which means that everything needed to reproduce a training run is available, including training scripts and data (although, in practice, one might not have the resources to do so)

Conventionally, the most capable foundation models have been closed-source models, although the gap has closed recently. Open weights and open source models are most relevant to composability: they can be easily integrated into a larger model and adapted–fine-tuned–for downstream use cases, in particular for science.

A specific example: Llama

To make things concrete, let’s look at a specific model, Llama 3.1-405B-Chat. This is an open weights large language model (LLM), tuned for chat, trained and released by Meta in July 2024. It is, at the time of writing, one of the most performant open weights large language models out there. Let’s see how it stacks up to the foundation model checklist:

• It’s pre-trained to perform next-token prediction on text from the internet: predicting what comes next in a sentence

• It’s fine-tuned to be useful as a chat model in several rounds, using a combination of supervised finetuning (SFT) and direct preference optimization (DPO)

• It’s trained on 15.6 trillion tokens, a significant subset of the internet. Writing this text would take a single human tens of thousands of years.

• It’s trained on up to 16,000 H100 GPUs for a total of 39.3M GPU hours, which on the open market would cost about 80M$.

  • This is only part of the total cost of the project. At 500 listed contributors, salaries likely add up to a comparable amount to the compute cost.

• It contains 405 billion parameters, which is far larger than what will fit in a single GPU’s RAM.

• It’s based on a transformer that is largely unchanged from previous generations of the same model

• The hyperparameters of the model–the number of training tokens and weights–were tuned to maximize performance for a fixed training budget by training small-scale models, computing scaling laws and extrapolating.

  

• It displays strong results on downstream tasks including question answering, code generation, and multilingual reasoning

• Some of its variants include adapters to ingest images, video and audio. The image encoder is a variant of CLIP. It is pre-trained on 2.5 billion pairs of images and corresponding text before being integrated as a component of Llama and fine-tuned end-to-end.

Already, we see several common characteristics of foundation models compared to more conventional, smaller-scale deep learning models that will be relevant to ethical discussions:

• They require large-scale engineering efforts to train from scratch, often far larger than what is available in a single academic lab

  • Academic actors may, however, be in a good position to perform finetuning or evaluations of models which have been released as open weights

  • The National Deep Inference Fabric, spearheaded by David Bau, is experimenting with offering free compute to academics trying to figure out what’s going on inside of large-scale models.

• Pre-training data is critical for the operation of these models. However, the sheer scale of the pre-training data makes inspecting the raw data challenging.

  • There have been high-profile instances where very problematic data were part of training sets, including CSAM in the case of the LAION-5B image set.

• Evaluation of model capabilities and limitations is non-trivial, as one-dimensional downstream metrics don’t capture the full breadth of what is in the models

• Because of their sheer size, it’s challenging to understand why foundation models work and to attribute their predictions to specific architectural components or data

  • There is an entire subfield of interpretability, including mechanistic interpretability and representation engineering approaches, that seeks to find mechanistic explanations for how the models work and to move them toward more helpful, less harmful behaviour. It’s become fashionable to release interpretable decompositions of these models to try and figure out what’s going on inside (e.g. the recent Gemma Scope from Google Deepmind).

• Composability, the wide availability of engineering tools and plentiful capital means that the field evolves very rapidly, and it can be hard to keep up

  • Blogs, newsletters and podcasts such as AINews (technical) or One Useful Thing (by UPenn prof Ethan Mollick, but aimed at a generalist audience) can help cut out the noise compared to the firehose of arXiv and conference papers.

What’s a foundation model for neuroscience?

Neuroscience is no stranger to the excitement around foundation models, and there have been several papers moving in that direction. I’m not going to make a strict distinction between foundation models and conventional machine learning models in neuroscience for this discussion but rather will focus on models which check off a large subset of the foundation model checklist.

We can broadly categorize these models in terms of which specific aspect of the relationship between an organism, its brain activity and its environment it attempts to model. Consider this (slightly Fristonian) diagram:

The world determines the sensory inputs of the subject; the sensory input exogenously drives ongoing neural activity; the neural activity and the mechanics of the body determine its position; and the body affects both the world and the sensory input of the animal. All the while, we can take measurements of brain activity, either invasively or non-invasively.

Many models perform a kind of graph surgery focusing on one or more nodes of this loopy graph, depending on the end use case2. Keeping in mind that these categories are fluid–e.g. an encoding model can easily be turned into a decoding model via Bayes’ theorem–it is nevertheless helpful to distinguish between these scenarios:

• Encoding models :: sensation → measurements.

  • Data-driven models focus on learning the mapping between sensation to measurement from scratch. When trained at scale on single neuron data, these models are sometimes referred to as digital twins, since they can be used instead of the real brain for virtual experiments (e.g. work from Alex Ecker, Andreas Tolias and many others [1], [2], [3]). This is most relevant where invasive data is plentiful, and inputs can be easily controlled, e.g. in vision.

  • Goal-driven or task-driven models focus on using AI models–often foundation models–trained on specific tasks, and mapping them to or comparing them to the brain. This has been popular not only in the context of vision but also in audition and language (e.g. Dan Yamins, Niko Kriegeskorte, Alex Huth, Ev Fedoronko’s work). Generally, the starting point has been models trained at large scale on AI tasks for other purposes, for example, image recognition. On occasion, models are trained from scratch, though they generally fall short of qualifying as foundation models (e.g. Mehrer et al 2021, Mineault et al 2021).

• Modelling the distribution of measurements :: measurements

  • Models can be trained to model the distribution of functional measurements, e.g. spikes trains, calcium imaging, EEG, MEG, fMRI, etc. Models such as LFADS are trained to compress neural data into smaller dimensional latents, which helps denoise and interpret neural data3. Masking and next-token prediction can also be used to find good latent representations without compression. Such models can be used for co-smoothing (e.g. predicting what a held-out neuron’s activity should be), or, in the same style as generative pre-trained transformers (GPTs) for text, for autoregressive generation of future neural activity.

  • Measurements can extend to complex structural data, such as transcriptomes, connectomes or imaging data.

• Decoding models :: measurements → sensation | behavior.

  • Decoding models have long been used in the context of brain-computer interfaces, to infer brain states, or to decode images from the brain. The advent of powerful models which find good latent representations of brain measurements has made these models far more powerful. When these models are applied to the decoding of structured sensory modalities (e.g. vision or speech), they can paired with powerful generative AI foundation models, leveraging their modularity. Striking examples have been shown, for example, of decoding seen images from fMRI, or decoding attempted speech from a locked-in patient in ECoG.

• Models of behaviour :: behaviour

  • Models such as DeepCutLab–based on a foundation vision model, and fine-tuned to track specific animals–have become an integral component of investigations into neuroethology. Extensions to multi-animal tracking, as well as models which can work in the wild on humans, have extended the range of behaviours which can be tracked, quantified and compared.

Many excellent reviews have covered some of these topics:

• Doerig et al. (2022): An overview of task-driven neural networks and the overall field of neuroconnectionism

• Wang and Chen (2024): An overview focused on generative decoding models based on foundation models

• Monosov et al. (2024): An overview of ethological approaches to computational psychiatry

Here, I’ll focus on the problem of modelling the distribution of measurements, which has only recently become possible at scale, with promising results. I have not been able to track down a recent review of these models, but I do keep a running table of the SOTA here (may be outdated by a few months).

An illustrative example: Towards a “universal translator” for neural dynamics

A paper released by Yizi Zhang and my former colleagues at Mila and Georgia Tech a few weeks ago illustrates some of the key technical components involved in learning the distribution of measurements–here, the distribution of spikes from electrophysiology measurements in mice, from the International Brain Lab (IBL) dataset. The IBL dataset contains large-scale Neuropixels recordings from multiple brain areas–thalamus, hippocampus, and visual cortex–while mice perform a decision-making task. This is a very rich dataset containing hundreds of hours of recordings. The authors here focused on a subset of the IBL dataset containing multiple recordings from the same areas, totalling 39 mice and 26,736 neurons.

The authors propose a pipeline to infer masked neural activity based on unmasked activity. Of note, to learn different useful downstream tasks–inferring the response of an unrecorded neuron, or region, or to predict neural activity in the future–the authors propose to train on multiple pretext tasks simultaneously (Tasks 1-4 above).

It’s worth it to stop for a second and ponder the nature of the modelling task to be performed. "Learning the dynamics of neural data” is a task that’s traditionally been tackled with unsupervised learning. For example, the now classic LFADS model uses a variational auto-encoder to learn an RNN which models the dynamics of neurons at the single session level. Unsupervised learning of that sort can be very efficient, but it does have some disadvantages: powerful generators can ignore learned latents, samples from the model can be over-smoothed, and tracking the progress of the model throughout training can be complicated (see e.g. the work on VQ-VAE to learn more about some of these issues and can be mitigated).

By contrast, masked autoencoding turns an unsupervised problem into a supervised learning problem, which is straightforward to train and scale. The model which infers one part of the input from another part of the same input can be pretty much anything one desires–a CNN, an RNN, a transformer, etc. The Universal Translator uses a version of the Neural Data Transformer–NDT– a straightforward scheme proposed by Ye and Pandarinath (2022). Specifically, it uses a variant of the NDT architecture which can handle multiple sessions–namely, NDT-1-stitch.

At the heart of transformer models of neural activity is a patching scheme: a method to transform neural activity into a set of discrete tokens which can be operated on by transformer layers. NDT-1 uses a simple patching scheme, where one time bin corresponds to one token4. First, neural activity is temporally binned; the original NDT paper uses 10 ms resolution, while the Universal Translator paper uses 20 ms. It’s then projected using a session-specific embedding matrix onto a fixed dimensional representation. Thus, a trial of length 2s gets transformed to 100 or 200 tokens. Standard transformer layers perform all-to-all attention to iteratively reformat this data. At the final layer, the tokens are unembedded to be used for the pretext task, prediction of masked neural activity. The Poisson loss is used to measure the mismatch between the predictions and the masked data.

Session-specific embedding matrices allow different datasets with different numbers of neurons to be handled by the same, fixed-size model. They also allow the model to implicitly align neurons onto the same latent space, “stitching” recordings together. Aligning different neural recordings together is a key technical challenge in these models, because it’s very unlikely that we ever measure the exact same neurons and voxels in different subjects.

The learned model can be used in several different ways:

1. It can be used as is to perform the same tasks it was trained on: predicting neural activity from one set of neurons to another, one set of areas to another or from the past to the future.

2. The learned model can be used to perform different tasks than the one it was trained on. One might freeze the learned model and put a logistic regression on top of the latent activity of the model to perform some downstream decoding task. Alternatively, one might put a (linear) decoder on top of the model and finetune the entire model end-to-end.

Here’s the model’s performance on the two sets of tasks. On the left four plots, we see the performance of the model on the same masking tasks the model was trained on. On the right two plots, the model is used to predict the mouse’s choice on an individual trial and the whisker motion energy during that trial. A baseline using less sophisticated masking is shown at the bottom.

These plots showcase typical results from a foundation model in neuroscience:

1. The model gets better as it is trained on larger and larger amounts of data (scaling laws)

2. The model’s performance is not restricted to the tasks it was originally trained on: it can be fruitfully finetuned for downstream tasks which leverage the latent representation learned by the model.

Or, as the authors state, “The performance of our approach continuously scales with more training sessions, indicating its potential as a “universal translator” of neural dynamics at single-cell, single-spike resolution.”

More is different

In the past, much of neuroscience has been concerned with the analysis of small-scale data, under controlled conditions, in a hypothesis-driven manner. Sophisticated large-scale recording technologies, including Neuropixels, together with archives to store large-scale datasets, many funded by the NIH–DandiHub, OpenNeuro, DABI, etc.–have led to an explosion in the size and breadth of publicly available datasets. Realizing the value of these extant datasets requires sophisticated tooling, engineering and methods: stitching together heterogeneous datasets often not designed with foundation models in mind.

I’ve had several discussions with hypothesis-driven-minded scientists who deride these efforts as “mere engineering” at best, or as “data dredging” at worst. I’ve had an equal number of discussions with professors who are worried about being left behind with the spread of AI methods which are inaccessible to most cash-strapped labs, a view reflected eloquently in a recent wonderfully titled editorial: “Choose Your Weapon: Survival Strategies for Depressed AI Academics”.

An optimistic viewpoint is that neuroscience is bottlenecked by a lack of tools, and foundation models distilling large-scale datasets are one such set of tools that might be useful for neuroscience. One of David Hubel’s core contributions to science, predating and enabling his Nobel-prize-winning studies of the visual cortex, was the invention of the tungsten electrode (Hubel, 1957). Tools tend to get commoditized; large language models are both far more powerful than predecessor natural language processing models, and far more accessible to the general public.

Here I show, in a rapid-fire sequence, some of the promise of recent efforts in foundation models for neuroscience, keeping in mind that the promise has yet to be fully realized. In particular, I want to make the case that more is different; that bigger models trained on larger scale data in neuroscience can display qualitatively different behaviour than smaller models.

Decoding sentences from EEG

Sato et al. propose to decode spoken sentences from EEG. Nothing unusual about this setup, except that they do so in a single subject from whom they recorded 175 hours of data.

The subject read sentences from a corpus displayed on a computer screen. They shaved the head of the subject (with dozens of sessions, washing gel out of hair becomes a bottleneck!). They trained a decoder composed of several pre-trained off-the-shelf models bundled together and fine-tuned end-to-end. They were able to reach 48% top-1 accuracy in a difficult 512-sentence classification task.

This is far higher than most people would expect from EEG, which has a reputation as being too noisy to be useful. To be clear, it’s not yet practical–who wants to calibrate an EEG headset for many months?–but it does show that decoding useful data from non-invasive signals might be feasible, especially if the representation generalizes across subjects.

Decoding gestures from EMG

On the other end of the spectrum, Meta Reality Labs (my former employer many moons ago) has released a paper showing their ability to create good generic decoders of wrist-worn EMG by training on lots of subjects. Notably, each subject was recorded for a reasonable amount of time (a little more than an hour).

Notice the x-axis in these plots: the number of training participants is in the thousands. While this is clearly outside of the purview of a single academic lab, forecasting performance from a smaller number of participants via extrapolation of scaling laws might be feasible (see, e.g. Ruan et al. 2024).

Decoding images from the visual cortex

Several recent papers propose combining learning good latent representations of fMRI activity and powerful diffusion models to decode images from brains, which I covered previously on xcorr.

MindEye 2 showcases that it’s possible to reconstruct images from the visual cortex with as little as one hour of fine-tuning data from that specific subject.

This model makes use of all the tricks we’ve discussed previously–powerful pre-trained models with specialized bridges that are fine-tuned on large-scale data to enable generalization across subjects. The results are visually striking–although one has to be careful not to over-extrapolate from the visual results how well the model does.

Predicting neuroscience results from text

The previous papers make it clear that powerful brain decoders can be built by learning the latent structure of neural data through supervised and unsupervised learning. Foundation models can stitch together large-scale information in ways which exceed the information-processing capabilities of a single human.

Analogously, large language models can integrate information from large-scale data and identify subtle patterns which are difficult to find in single texts. An impressive recent demonstration is Luo et al. (2024), “Large language models surpass human experts in predicting neuroscience results”. They measure the ability of LLMs to correctly identify real neuroscience abstracts from counterfactual abstracts.

They assembled a dataset of abstracts published in 2023 in the Journal of Neuroscience and created synthetic versions of the same abstracts with the valence of results inverted. E.g. replacing suppresses with enhances, reduces with increases, etc. They then measured the likelihood of the real and fake abstracts according to pre-trained LLMs, directly using the perplexity measure that these models use to predict the next token. They found that LLMs could indeed identify the correct abstracts more than 80% of the time, despite not having been trained on these abstracts5, and that this was far higher than human experts (neuroscience PhD trainees and profs without access to a search engine). Base models performed better than chat models, probably related to chat fine-tuning hurting the calibration of these models; and larger models performed better. Of note, the output of the models was not a simple yes/no answer, but rather a continuous variable (perplexity over the abstract, in units of nats) whose absolute value expresses the certainty of the model and correlates with human judgement.

The resulting model could potentially be used in several use cases, including identifying controversial hypotheses that should be investigated or, conversely, identifying latent hypotheses which are likely to have already been confirmed in the literature. This paper is one example of a recent trend, LLMs for neuroscience, which was recently covered in a review from Danilo Bzdok and colleagues.

It’s not that easy

You might come away from this tour with the impression that one can throw a lot of data and compute into a high-capacity model and that’s the end of it. Of course, it’s quite a bit more subtle than that.

Foundation models are still susceptible to shortcut learning, where decisions are made based on accidental and brittle correlations. We can easily illustrate this with a ResNet-50 trained on ImageNet. Characteristic features of the category tench, a freshwater Eurasian fish, include human fingers, faces, and camouflage clothes. Here, the shortcut learnt relates to sports fishing: consequently, identification of the animal in its natural habitat is impaired. Similarly, the EEG speech decoding study almost certainly leveraged EMG artifacts from the jaw muscles generated during overt speech to boost performance, as opposed to decoding only neural activity, although they did bound the extent to which this happens using a synthetic data analysis.

“Universal decoders” can fail if they’re not trained on representative samples of the population; the Meta team did a good job of recruiting a demographically representative population, but suffice it to say this required a lot of resources that are often inaccessible to single academic labs.

Powerful generative decoders can fool human observers into thinking that they perform better than they actually do. When we judge the quality of a decoder, we might rely on superficial features, like the aesthetic qualities of the generated images or speech, as opposed to how well-grounded the generated sample is to the neural data. Shirakawa et al. (2024) report that this type of generative decoding model tends to overfit to the distribution of the inputs, such that they fail to generalize out-of-distribution.

Large-scale models are only as good as the data that’s fed into them: garbage in, garbage out. LLMs can fall for imitative falsehoods, repeatedly debunked falsehoods in the raw dataset, unless they’re carefully fine-tuned away. These imitative falsehoods, which can be detected using benchmarks such as TruthfulQA, can range from the benign–e.g. the idea that cracking one’s knuckles causes arthritis–to the harmful–election denial and mischaracterizations of the efficacy of vaccines. In the same way that wrong or incomplete neuroscientific hypotheses can have a long shelf life through repetition, for example the chemical imbalance theory of depression, large-language models can fall for oft-repeated but scientifically dubious “facts”.

Finally, large-scale training is not a panacea for all problems. The accuracy on downstream tasks depends not just on the size of the training sets, but also on the accuracy of the labels, the dimensionality of the signal, and the signal-to-noise ratio. Schulz et al. (2022), for example, extrapolated from current datasets in brain-imaging-based phenotyping to estimate how well these methods would work when tested on a million subjects.

The results were sobering. Even at a data scale of a million subjects, which could easily cost a billion dollars to acquire, MRI-based classification accuracy was projected to reach roughly 0.6 for depressed vs. not-depressed, barely above chance. Scaling laws can be cruel, with performance improving only log-linearly with dataset size.

Foundation models for neuroscience: opportunities and pitfalls

We can view foundation models in neuroscience as a logical extension of an ongoing trend: training larger models on ever larger datasets. A basic mental model for the current trends is that it’s simply more of the same: more data, more parameters, and higher accuracy. Previous assessments of the opportunities of the machine learning models in neuroscience and their pitfalls, in this view, are simply amplified, both positives and negatives.

I’ve taken a slightly different view here, which is that bigger models can be qualitatively different:

• Foundation models are composable, which enables more rapid progress than training from scratch for each task

  • Foundation models can be used for multiple downstream tasks–even unanticipated tasks–via a simple process of fine-tuning. It moves us from a world of bespoke models to a library model, e.g. LEGO bricks.

  • Although training from scratch is too resource-intensive for many academic labs, downstream uses don’t require nearly as much compute

• Foundation models can take advantage of unlabelled data, which is far more plentiful than labelled data. Hundreds of thousands of hours of data are stored in public archives such as DANDI, OpenNeuro and DABI.

  • Although metadata can be of varying quality in these datasets, LLMs can be used to assist in metadata extraction and filtering.

• Foundation models have been proposed for data formats which have resisted conventional large-scale machine learning, e.g. graph data, spikes, transcriptomics, etc.

• Foundation models are somewhat of a misnomer–the models themselves, transformers et al.–don’t matter nearly as much as the data that’s used to train and fine-tune the models. This data-centric view of AI is a mindset and tooling shift from the previously prominent model-centric view.

I’m excited about the applications of foundation models in both discovery and applied neuroscience. Powerful, off-the-shelf and accessible tools have the potential to accelerate discovery in neuroscience, by making the value in existing datasets visible. They also have the potential to accelerate applications, for example in the context of brain-computer interfaces, making use of rich data to make invasive interfaces more data efficient and noninvasive interfaces more accurate.

Foundation models, like their more specialized single-use machine learning predecessors, when used in sensitive human health-related applications, have the potential to amplify societal biases, deplete autonomy and agency, and raise important privacy concerns. These concerns, which also exist for AI systems at large, have been the subject of several excellent books (e.g. The Alignment Problem). It’s likely that existing ethical frameworks for machine learning in neuroscience–e.g. here and here–already anticipate and can accommodate many of the consequences of foundation models.

The question that I will try to address during the NEWG meeting is whether there are unique risks which are posed by foundation models over and above conventional models. I’m a technical person, not a bioethicist, so I tend to think of risks in technical rather than societal terms, caveat lector. There are three unique categories of risks that I foresee.

The first risk comes from the inscrutability of models trained at a large scale. It takes resources and diligent work to fully characterize the model’s biases across a wide range of downstream use cases; to suss out instances of shortcut learning; and to visualize all the data used in training. It’s largely thankless work that’s poorly incentivized in conventional academic environments–that is, it maps poorly to papers. It’s also not necessarily something that scientists are trained to do.

This could be addressed, however, by novel funding mechanisms specific to tooling, by the hiring of data scientists and engineering, as well as by building training courses on the uses and misuses of large-scale models. Because foundation models are so data-centric, projects which are organized around the creation of high-quality datasets and tooling are highly relevant. We’re seeing a Cambrian explosion of innovative non-profit focused research organizations (FROs) and large-scale academic projects that scale neuroscience data collection and foundation models for science, like e11bio, Forest Neurotech, FutureHouse, and the Enigma project.

The second, related risk stems from the paradox of automation: as more capable machine learning systems get integrated into decision-making, we trust them more, and become less capable of correcting their mistakes. One (rather naive) example is in automated seizure detection: as machine learning-aided detection becomes broadly more useful, less time is spent on examining the raw data, fewer practitioners are competent in reading the raw data, and mistakes take longer to be noticed and corrected. Human-AI teams tend to cluster into either the human doing all the heavy lifting (when the AI system works poorly) or the AI doing all the work (when the AI system is effective).

The third is around equitable data access. Foundation models are both data hungry and valuable. The prospect of transforming sensitive neural data, collected through neurotechnology, into intellectual property should give one pause. This raises critical questions about data ownership, privacy, and the potential for exploitation.

I do think that despite these challenges, these risks can be mitigated with the right regulatory and funding environments. The potential of foundation models in neuroscience to accelerate the discovery to clinical translation pipeline is truly exciting. These models offer unprecedented opportunities to analyze complex neural data at scale, potentially uncovering patterns and relationships that have eluded traditional methods.

As with the rest of the Neuromatch courses, it’s focused on practical computational concepts given in an interactive tutorial format through the Colab platform. Students in small virtual pods go through the course accompanied by a TA, exploring the state-of-the-art in neuroAI. Neuromatch is a 501(c)(3) non-profit dedicated to making science education more accessible and equitable. So far, 10,000 students from over 101 countries and territories have gone through the CN, DL and computational climate science courses.

What is neuroAI?

Our fearless course director Xaq Pitkow had a mammoth task: to define a burgeoning field that resists a short definition. Through many iterations of the curriculum, we’ve come to a story that I think is broad, inclusive and narratively satisfying.

NeuroAI is the study of the common principles behind natural and artificial intelligent systems. There are two main strains of practice in neuroAI:

• The neuro → AI route: learning principles from brains that can be applied to create more robust and efficient artificially intelligent systems. This subfield is often known as neuro-inspired AI.

• The AI → neuro route: taking inspiration from advances in artificial intelligence to understand natural intelligence. This subfield has also been referred to as neuroconnectionism.

These threads of research have ancient roots. Homer, in the Ilyad, describes the story of Hephaestus, god of metallurgy, who built a set of artificially intelligent agents (robots) made of gold to assist him in his workshop, in part to relieve him of physical limitations brought by his limp. Understanding biological intelligence–knowing oneself–is a theme that runs from classic Greece to ancient Vedic and Taoist traditions, with the first medical investigations of the brain documented in an ancient Egyptian papyrus dated 1600 BC.

The fact that these roots are so deep hints that both neuro and AI are tapping into deeply human desires to understand and build. One of the founding principles of the course is to instill a radical curiosity in students toward the goals of other disciplines. That means immersing oneself in the language and social context of both these disciplines and committing to read the primary literature from both sides. This is the key to bridging the gap between AI and neuroscience.

Generalization and inductive biases

We could have told many stories about natural and artificial intelligence to build a coherent narrative: phylogenetic refinement vs. the artificial evolution of AI systems; prediction and prospective learning; optimization of behaviour under resource constraints. Each of these contains a facet of truth about neuroAI, and one could create an entire course around each of these ideas.

I really like where we ended up with the overarching narrative: intelligent systems generalize. Intelligent systems must be able to adapt to changing circumstances–including circumstances previously unseen–rapidly and effectively. Because of the no-free lunch theorem, generalizing to arbitrary circumstances is impossible; it is only by having strong inductive biases that we’re able to generalize beyond our immediate experiences.

Starting from that premise, the course covers a wide range of topics from basic neuroscience all the way to theoretical deep learning, from synapses to symmetry. It’s a wide scope, and recognizing that a lot of the questions in our field are unsettled, we build each day in a T-shaped structure: we go broad in the intro lectures, and go deep in the weeds in the tutorials. We always tie back the days to the question of generalization. Our goal was to give students from all backgrounds the ability to read other literatures through a dense map of concepts tying the two fields together, and to continue on their learning journey far beyond the end of the course.

Course structure

There are 9 days of content, spread over two weeks:

• Overview of neuroAI

• Comparing tasks

• Comparing networks

• Micro-architecture

• Macro-architecture

• Cognitive structures

• Micro-learning

• Macro-learning

• Mysteries

Students get to practice these concepts deeper in a set of projects led by the amazing Eva Dyer.

Sign up now

Applications are open from March 1st until March 17th (for TAs) and March 24th (for students). Course tuition is adjusted according to local cost-of-living to be affordable for all. It’s a great occasion to be part of the inaugural class for what we hope will be a foundation for the next generation of neuroAI scientists.

We are grateful to ARNI, the NSF AI Institute for Artificial and Natural Intelligence, for their support.

The idea behind WBE is appealingly simple: if you can simulate a human brain in silico in excruciating detail, you’ve got an AGI. Simulate it faster, fix the “bugs” (e.g. lower plasticity in later adulthood) and you have faster-than-human general intelligence. This field has seen cycles of hype (e.g. TED talks 1, 2) and disappointment, as captured in the wonderful film In Silico. We’re currently past a cycle of hype and have seen some real results coming from the field of connectomics.

Hi-fi paths toward WBE

Slice the brain using electron microscopy, use petascale computing to label and stitch together the slices, and you have yourself a connectome. The MiCRONS project has done this for a millimetre-cube of cortex in mice; the fly connectome has already revolutionized the study of these organisms.

Provided a connectome for a whole brain, it’s possible to simulate the model forward, using single-neuron models of varying biological plausibility, from linear-integrate-and-fire to detailed multi-compartment models with Hodgkin-Huxley dynamics. Simulate the body and environment, in addition, and you have yourself a whole-brain-body-environment simulation. We’re not there yet, but one could imagine this could happen in the not-too-distant future, especially for smaller organisms.

How well would this work? It depends on the answer to a few empirical questions. There was a recent article from Nicole Rust in The Transmitter magazine on whether the brain is chaotic, that is, whether it displays exponential sensitivity to initial conditions. If the brain displays chaos, small measurement errors in initial conditions will propagate to arbitrarily large errors over time. It’s likely that if the brain is chaotic with respect to neural activity, then it will also display exponential sensitivity to the weights of its connection matrix. Chaos is incompatible with literal whole-brain emulation, in the same way that you can’t literally predict the weather over longer periods than a couple of weeks–though you can predict the climate.

There’s also the open question of how much the static connectome can capture. Dendrites and synapses roll over; we can’t always infer the strength of a connection from its static shape; there’s ephaptic coupling and non-synaptic transmission. The latter appears to be a bit of a bottleneck in C Elegans, where recent results show that the connectome is not very predictive of neural activity.

Lo-fi paths toward WBE

When people talk about WBE, they generally refer to the connectomics route, followed by single-neuron emulation. Indeed, Anders and Bostrom1 (2008) wrote a roadmap for WBE that looks a lot like modern EM-based connectomics. What’s interesting is how they cast a formal definition of whole-brain emulation, in a footnote in the introduction:

Their definition is based on accurately predicting the future state of a brain starting from an initial state. This dynamics-based definition of whole brain emulation looks a lot like time-series forecasting or next-token prediction, in the style of GPT. With modern artificial neural networks (ANNs), direct time-series forecasting without reference to a connectome looks like an increasingly feasible way to achieve something that looks like WBE.

Furthermore, there is flexibility in how one defines the state. A behaviourist might focus on externally measurable variables only, e.g. the activations of all the muscle afferents of the body as well as the position of the limbs. Someone inspired by Mountcastle might care primarily about the average activity neurons in cortical columns or in mini-columns. In those cases, one reduces the state from trillion-dimensional (i.e. 86 billion neurons times all the state variables in each neuron) to anywhere from a few hundred to perhaps a billion. Bringing down the dimensionality by several orders of magnitude makes these approaches seemingly more tractable over the short term.

Several people in these Foresight Institute talks refer to these approaches as lo-fi, to distinguish them from more conventional high-fidelity, single neurons+connectome approaches. I wasn’t able to track down a written reference that uses that term–it might be a jargon term associated with that Institute, but I think it’s useful enough that I’ll use it here.

Catalin Mitelut categorizes lo-fi approaches into three buckets:

1. top-down (i.e. behavioural cloning approaches)

2. bottom-up (i.e. neural activity cloning approaches)

3. hybrid approaches

Top-down (or outside-in) approaches

Say that you:

1. measure behaviour in an animal or a set of animals with high accuracy

2. learn good generative models of it

3. embody it in a virtual body

You might eventually be able to create a virtual agent with all the same capabilities as animals of that species. This is, in theory, no different than generating handwriting, text, or images. Setting aside (dreaded) questions about philosophical zombies, I think this is quite feasible along several different axes.

Collecting behavioural data is easier than collecting neural data. Deeplabcut and related approaches have made behavioural tracking feasible with off-the-shelf tools. Virtual bodies are getting better thanks in large part to the video game industry (see video above for an example). There is abundant data of playing video games, e.g. Minecraft.

Furthermore, training an artificial neural network in silico on a behavioural task often leads to solutions like the brain’s [previous coverage on xcorr here, here and here; although see here for a counter-example]. Thus, this purely behaviourist approach might find representations which converge to the brain’s.

Aside: metrics

An important question is how to measure the capacity of agents that work on the principles of behavioural cloning. The naive approach of the whitepaper, as noted by the whitepaper itself, doesn’t work if the agents are chaotic; after a certain time horizon, prediction becomes impossible in principle. Varying Turing tests have been proposed to measure the capacity of agents:

1. Navigation Turing test: “predicts human judgments of human-likeness” in the context of a navigation task

2. Embodied Turing test: “challenges AI animal models to interact with the sensorimotor world at skill levels akin to their living counterparts”

Now, we could use next-token prediction in the style of GPTs for optimization, but we can’t use that for evaluation over long horizons. In addition, that metric, while useful for pretraining, is not aligned to the final desired outcome. Ultimately, we will need to choose what kinds of behaviours are considered equivalent, or what kinds of equivalence classes are defined by our chosen metric. This papers proposes to use distributional alignment rather than MSE or cross-entropy for long-horizon behavioural prediction.

Bottom-up (or inside-out) approaches to WBE

Mitelut also documents bottom-up approaches to whole-brain emulation. Here, rather than cloning the behaviour of an organism, one clones neural activity, i.e. the electrophysiological activity of neurons. Konrad Kording and colleagues are proposing to use causal manipulation of neural activity in C Elegans to fully reverse engineer a worm’s brain, thus sidestepping the issues translating the connectome to neural activity. But could you scale this to a mouse or a man?

Mitelut points out that, extrapolating from current trends (e.g. Stevenson and Kording 2011), we wouldn’t reach human whole-brain recording capacity until the 2060s, by which time I will be retired.

That conclusion rests on a few assumptions, which I think are debatable:

1. There will be no breakthrough technologies in brain recording that change the slope of the current curve. People are trying to change that, e.g. Sumner Norman has a new startup using ultrasound for whole-brain recording, Mary-Lou Jepson is pushing forward with OpenWater, my old group at Facebook demonstrated 32X increase in SNR in diffuse correlation spectroscopy, etc.

2. You need to record the whole brain to have any chance at WBE via this bottom-up approach. If you believe the random manifold theory of brain coding, a random sparse subset should be sufficient.

3. You can’t stitch together recordings from multiple organisms together to obtain one equivalent amortized organism. Recent results from Matt Perich, Juan Gallego and co. show that different individuals from the same species learn equivalent latent manifolds to solve motor tasks, which means alignment should be feasible in theory. This has motivated some significant advances in neural stitching from my colleagues and I, which I think could change the game.

I agree in principle, however, that purely bottom-up approaches are hard to get working without any anchoring to behaviour.

Hybrid approaches to WBE

That brings me to a third set of approaches, which attempt to stitch together observed behaviour and neural data. There hasn’t been a ton of work in this area, but I’m quite bullish on the whole concept.

We’ve seen a few areas where training a particular ANN architecture, with the right inductive biases, on the right combination of tasks, leads to a trained architecture that has an excellent one-to-one correspondence between simulated and real units. This has been shown in the retina by Niru Maheswaranathan et al. (2023), which I mentioned a few newsletters ago. This is also apparent in work from Lappalainen et al. (2023), which trained a network with the exact connectome of a fly, derived from EM, with task-driven learning, to obtain a pretty solid model of a fly’s optic flow detection system.

Challenges and opportunities

From watching a number of the talks in this series, WBE–lo-fi and hi-fi varieties–is more feasible than I had originally pegged my estimate. It certainly seems a lot closer than the 40-50-year timeline that I’ve seen quoted (reference; derived from surveys of AI researchers), especially when it comes to lower-fidelity approaches. Are we likely to bump into insurmountable obstacles along the way?

The original whitepaper estimated that in 2019, a human-sized population of integrate-and-fire neurons could be simulated by a supercomputer that costs a million dollars. Are we close to the count? If we stick to a 1 exaflop estimate for a whole human brain, we could achieve real-time simulation with 1000 H100s2. That would cost more than a million dollars (perhaps 30M$3), but it’s not an infinite amount of money either, nor is it out of reach of the best-funded academic institutions. As a point of reference, Harvard’s Kempner Institute announced it would acquire 384 H100s.

If compute is not a limitation, then what is? If we require a whole connectome, we’re talking billions of dollars at current costs to acquire that data (it cost ~100M$ to scan 1 mm3 of mouse cortex in the MiCRONS program). If we assume that the cortical algorithm is universal–that we can infer universal connection rules from measurements of partial connectomes–then we might not need a whole-brain connectome. Lo-fi approaches based on distilling behavioural and neural data are cheaper still. How much brain and behaviour data do we need for lo-fi approaches?

Another open question is about reducibility. What is the purpose of a WBE model at 1/10th the (spatial|temporal|behavioural|neural) resolution? 1/1000th? What about coverage: do we get anything by simulating, with very high accuracy, a small chunk of cortex?

We’ve learned from the story arc of Henry Markram–the leader of the Blue Brain project who promised to send a hologram of himself to a TED conference by 2019. We have to have to carefully manage expectations about the potential benefits of simulating a significant chunk of the brain. We have to go several levels deeper to identify the downstream users of these technologies and find capabilities which are likely to be unlocked at different levels of resolution of these technologies. If we shoot for the stars and miss, can we make it to the moon? While the endgame may be a form of AI, the intermediate applications are likely to be in human health.

If we can make clear the link between the fidelity of a simulation and corresponding capabilities which get unlocked, we’ll have a much clearer technology tree benefitting both AI and neuroscience. This is an exciting project for NeuroAI in the coming years.

Theories of everything in neuroscience

Why are we here? What are brains for? What is intelligence? Given looming deadlines and an ever-growing to-do list, it’s hard to think deeply about these things for more than a couple of hours at a time. I decided to take some time over the holidays to think deeply about these issues.

I was excited to pour over A Brief History of Intelligence by Max Bennett. It was recommended by Dileep George, who wrote one of the blurbs, and more recently by Richard Sutton (of RL and Bitter Lesson fame). I’ve been enjoying this breezy, narrative history of intelligence. The author uses the history of early bilaterians, vertebrates, mammals, primates, and humans to craft a narrative around how cognitive capacities, neural structures, body plans and environments co-evolved to create adaptive behaviours.

One example of these ideas is our urbilaterian ancestor, a worm-like creature not too dissimilar from a modern C. elegans. This worm was a swimmer engaged in finding food and avoiding predators. This lifestyle encouraged the appearance of centralized, fused ganglia to maintain a consistent internal state; i.e. a brain. In a hungry state, the worm would be more explorative, even though this would mean it was more exposed to predators; if it was satiated, it would stop moving to conserve its energy. Various signals emerged to carry these states, of approach and avoidance, of stress and relaxation, ancestors to what we would eventually come to be called emotions. And with this short-term state could come adaptive behaviours, such that a stimulus does not always lead to the same response, depending on the state of the animal. Link the environment and outcomes and you can have reinforcement of particular behaviours. Thus, a sensorimotor loop, emotions, and learning were preserved and built upon in vertebrates to come.

The book is conceptually aligned with the idea of phylogenetic refinement as advanced by Paul Cisek. Phylogenetic refinement is a process where the brain's structure and functionality are shaped not only by individual development (ontogeny) but also by the evolutionary history of the species (phylogeny). Current brain functions have evolved from simpler forms present in ancestral species, and these functions have been refined over evolutionary time. In other words, nothing in biology makes sense except in light of evolution. It’s an example of a theory of brains with wide explanatory power.

I like this framing both from an explanatory point of view and as a roadmap to practically embed intelligence in artificial systems. An enterprising person might take a look at the functions or areas captured by these theories and instantiate them in a mechanistic model. reflexes, imitation learning, reinforcement learning, and social learning are distinct aspects of learning complex motor programs. One might want to instantiate principles of hierarchical motor control, as implemented by the brain, to build flexible robots that don’t require perfect actuators. We can see these principles at play, for example, in this demo from Deepmind of cheap robots playing soccer.

A handful of people have taken the logic of compartmentalized, phylogenetically refined models to cover a large array of functions and areas, notably the team led by Chris Elliasmith. Perception, working memory, hierarchical motor control and reward are linked in spiking neural networks which are fast approaching the size, in terms of distinct units and number of synapses, of large language models. These are not (yet) brains in a vat, and are less capable than LLMs of the same size, but they point the way towards models that bring together multiple capabilities, in particular in terms of closing the action-perception loop.

Theories of everything, everywhere, all at once

Your average undergraduate neuroscience class can feel an awful lot like describing a stamp collection: lots of facts disconnected from the bigger picture. I like these attempts at what may be called theories-of-everything (TOE) because they allow numerous phenomena to be described coherently.

When it comes to translating natural intelligence to artificial systems, aggregating and distilling information is a crucial intermediary. The specific implementations of neural algorithms in biological neural networks may not translate one-to-one to their artificial counterparts. For one, the engineering requirements are vastly different–server farms using the energy of small cities and humans running on bananas live in very different regions of SWaP-C (size, weight, power and cost).

4 years ago, I saved this tweet from Josh Vogelstein that listed some theories of everything in neuroscience. I have made some slow and steady progress in better understanding the list–I even managed to trudge through a bit of Stephen Grossberg’s book, despite dire warnings. Josh’s list has been a source of endless inspiration.

I will concur with Surya Ganguli that many TOEs in neuroscience and on the topic of intelligence in general are “not even wrong”, in the sense of not being falsifiable. One of the sharpest examples–from a slightly distant but related field–is Legg and Hutter (2007), who introduce a universal definition of machine intelligence. This general intelligence is the weighted average reward obtained by an agent over an ensemble of environments. The weighting is given by the (negative exponential) of the Kolmogorov complexity of the environments. Kolmogorov complexity is the length of the shortest computer program that defines the environment. That weighting is eminently sensible: simpler environments are more likely and should be up-weighted. There are also many more incompressible environments (e.g. generated by a universal Turing machine seeded by a random number generator) than there are compressible environments, hence we should upweight compressible environments.

The problem, of course, is that Kolmogorov complexity is uncomputable, as a consequence of the negative resolution of the halting problem. So, their scheme is both really enlightening and utterly impractical. Talk about not even wrong: it’s not even computable! This hasn’t stopped that paper from sowing the seeds of what would later become Deepmind and foreshadowing both modern RL and LLMs. A good theory of everything should highlight some interesting underlying ideas that can be sifted through by an enterprising and discriminative AI researcher, even if the ideas are not directly presented in practical form.

Closing out the year

Part of the reason I’ve been thinking deeply about the grand scheme of things is that I’ve taken up the role of curriculum lead for the Neuromatch Academy NeuroAI course. I’ve got big shoes to fill (Konrad Kording was curriculum lead for the DL course and Gunnar Blohm for computational neuroscience), but with a kickass team led by Xaq Pitkow I know we will pull through. I want to bring a spirit of radical intellectual curiosity of discussing really big ideas, while also getting into the weeds and teaching students how to implement these ideas.

Subscribe now

We have a pretty solid curriculum skeleton lined up and several day leads confirmed. If you’ve read this far, don’t be surprised if you get an email from me, Xaq, or projects lead Eva Dyer, to help out with the course. It’s an awesome opportunity to push our field forward and teach the next generation of neuroscience and AI weirdos.

As a parting thought for this year, I’m grateful that I get to think deeply about brains and AI for a living! I’ve had the amazing support of the Mila team and colleagues since I joined in June–we have some awesome things lined up in terms of building and releasing foundation models for neuroscience, which I’ll write about in 2024. It’s been thrilling to share my thoughts with you–growing to a bit over 500 650 subscribers in less than 6 months, a not insignificant proportion of all neuroAI researchers. Thanks for reading, and have a safe and happy New Year 🎉 🍾

I recently asked on X for suggestions for NeurIPS NeuroAI papers. You did not disappoint! I received several dozen replies referencing papers across the main conference and workshops. I realized I needed to be a bit more systematic about it. In brief:

• I fed all 3,600 papers from the main conference to GPT-4 turbo and extracted 197 NeuroAI and NeuroAI-adjacent papers.

• I compiled a list of the most relevant workshops.

How to survive NeurIPS

If you don’t want to get massively overwhelmed, I would suggest that you go through the list of main conference papers and focus on < 10 papers which are directly and highly relevant and timely to your work. Then, go see 2-3 workshops. Once the conference is past, you can draw down the papers throughout the year. Pace yourselves!

I don’t get a lot from being at the conference in person–between the 9AM start, the jetlag and bad coffee–and prefer to watch from the comfort of my own home at 1.5X with subtitles on. NeurIPS is one of the best conferences to watch from home IMO, as their tech is top-notch.

The main conference paper pile

I made a little visualization and interactive app in Airtable to allow you to browse all 197 NeuroAI papers from the main conference. You can also download the CSV if you want to do secondary analysis straight from the Airtable link.

It was a lot more papers than I anticipated! Our field is going strong! How I made this:

1. I went to the NeurIPS website and downloaded a json with all 3600 main conference papers, which I found using Cmd+Opt+I.

2. I fed all abstracts to GPT-4-Turbo to extract one of 5 broad categories of papers related to NeuroAI. I had 2 core categories, 1 neuro-data-analysis, 1 neuro-adjacent, and one “has nothing to do with NeuroAI” category. I re-used a script I made for the NeuroAI in Montreal conference, for which I was a co-organizer. Running this took 5 minutes and cost 30$ in OpenAI credits. I also asked GPT-4 for TL;DRs.

3. I manually went through the ~300 papers it selected in the top 3 categories to select the ones that I agreed were NeuroAI. I added in anything with the Neuroscience topic. I sorted the results such that anything anyone personally shared with me on X by Sunday night was shown at the start.

4. I embedded abstracts with a all-mpnet-v2 sentence transformer to find the nearest neighbours for each paper.

5. I dumped a CSV of everything and imported it into Airtable.

This took me about the time it would take me to read ~3-5 papers in detail. ALL the papers you shared with me on Twitter from the main conference were ALSO INDEPENDENTLY selected by GPT-4 and correctly bucketed into one of the 4 NeuroAI categories. These are truly the days of miracle and wonder.

The workshops

Here are the workshops that are neuro-heavy:

• UniReps: Unifying Representations in Neural Models

• Information-Theoretic Principles in Cognitive Systems (InfoCog)

• Associative Memory & Hopfield Networks in 2023

• Symmetry and Geometry in Neural Representations

These might contain some neuro:

• Gaze Meets ML

• XAI in Action: Past, Present, and Future Applications

• Generative AI and Biology (GenBio@NeurIPS2023)

Finally, don’t forget the NeuroAI social on Dec 13th (Wed).

Towards Monosemanticity: Decomposing Language Models With Dictionary Learning

Anthropic and the mechanistic interpretability team led by Chris Olah and Shan Carter have been working relentlessly to obtain mechanistic interpretations of deep neural networks. Some of their prior work decompose visual circuits in CNNs in ways that would make Hubel & Wiesel proud. There’s a lot of neuroscience inspiration in mechanistic interpretability, because neuroscientists have had a 50-year head start in finding ways of opening up the black box of (biological) neural networks. I really enjoyed this paper: it has some interesting ideas and lessons for neuroscientists interested in sparse vs. dense coding.

This work follows from an ongoing work thread on understanding transformers trained for natural language. Previously, they had found that a transformer trained on this task represents sequences densely. That means that a single neuron doesn’t have a lot of meaning, and there is no sense in which neurons form a privileged basis: a single neuron in the network is bafflingly hard to understand. It might participate equally in coding for base64, Arabic, rhyming patterns, etc. They previously proposed to resolve this by adding in a dash of sparse coding: adding a softmax in the middle of the transformer to force activations to be sparse. With this simple modification, they found that the single-neurons in the network were far easier to understand, with sparse coding causing representations to be disentangled.

This time, they threw out their old method in favour of a different way of implementing sparse coding. They realized that training one network directly with a sparse coding objective was unwieldy, and that things trained more stably with two networks. The first network learns to predict the next token, without constraints on the distribution of the neural activity; the second does a sparse coding decomposition of the MLP branch of the first. In terms of their actual implementation of sparse coding, it’s a two-layer MLP with an L1 loss that is trained in a supervised fashion, nothing too esoteric.

The directions found by the sparse coding algorithm are far more interpretable than the dense representation of the original network: units correspond to base64, Arabic, parts of HTML, etc. Furthermore, they find evidence of finite-state-machine-like representations, such that, e.g. they can trace the kinds of state tracked by the network as it autocompletes HTML strings.

Links to neuroscience

There’s a longstanding debate in neuroscience regarding sparse codes vs. dense codes. The last ten years have advanced the view that individual neurons don’t really matter; in one view, popular in motor neuroscience, neurons are random projections of the underlying dynamics (e.g. Gallego et al. 2017; Ebitz & Hayden 2021) .

I’m personally not fully on the side of reified dynamics–I think single neurons are important, which is mostly a reflection of my background as a visual neuroscientist rather than motor neuroscientist. Off-axis coding may be less efficient in biological neural networks as a consequence of the statistics of Poisson processes. I do think it’s been an important correction to the previous single-neuron-at-a-time view. Maybe the brain has figured out how to do (partial) sparse coding directly; or it may have mixed dense/sparse coding in the style of the two networks the Olah and co. Their work highlights that it’s not trivial to get sparse cording networks learning effectively, and we should spend some time as neuroscientists to investigate that.

On a more pragmatic note, I think we tend to fit toy models in neuroscience in the hopes that they will be interpretable. The team at Anthropic proposes a different approach–learn a giant and hopelessly complicated model that models the phenomenon perfectly; then learn a simpler model to explain the outputs of the complicated models. One advantage is that the complicated model can be infinitely probed. Indeed, they probe on billions of examples. The task of the second network–which distills the continuous outputs of the original outputs, similar to the problem of knowledge distillation–might be harder than the task of the original network. Splitting the problem in two untangles the two issues in an approachable way. Interestingly, their efforts show that they can extract information that looks a lot like neural ensembles and cellular assemblies. Perhaps the way to understand the language of the brain is to learn high-capacity proxy models which are then taken apart.

We see the filiation of this work from sparse coding neuroscience to AI; in fact, Bruno Olshausen is a reviewer of this paper. We could imagine that a lot more of the machinery to interpret biological neural networks could be useful going forward in mechanistic AI interpretability–from reverse correlation and multi-perturbation Shapley analysis to tensor component analysis.

Identifying Interpretable Visual Features in Artificial and Biological Neural Systems

An embedded problem in the previous paper is finding interpretable bases for complex data. They lucked out with sparse coding, but there’s no guarantee that sparse coding yields interpretable insights; it just so happens that in this case it does. Ultimately, “interpretable” can only be really understood in light of the limitations specific to the architecture of the human mind–although people have tried to find non-human-mind-linked measures of complexity, e.g. Kolmogorov complexity, they are intractable.

So how do you search basis space specifically for interpretable directions? Klindt, Sanborn and colleagues propose some interpretability metrics in visual classification networks based on color consistency, LPIPS and label consistency. Basically, a direction is interpretable if the images that maximize activity in that direction are visually similar. Given an interpretability metric, they can then evaluate a specific direction, picked from an off-the-shelf algorithm like PCA, K-means, ICA, sparse coding, etc. They’re able to use this to find some directions which are more interpretable in coding space.

Similar to the previous paper, they find that off-axis can be far more interpretable than on-axis directions (individual neurons). I think it’s a great start and a solid attempt at writing down a classifier for interpretable vs. non-interpretable. I do have some qualms about their choice of interpretability metric; taking a page from Kolmogorov complexity and minimum description length, it seems like “how hard it would be to communicate visual concept X using language” would be closer to what they’re getting at. It would interesting to reimplement the same idea using vision-language-model-based captioning + compression as the metric to optimize.

Related: Tim Lillicrap & Konrad Kording (2019) on what it means to understand a neural network.

Thanks to Sophia Sanborn for sending me the second paper. Got something interesting to share? Send me a message and I’ll try my best to cover it in an upcoming post.

Workshops! ML doesn’t do reviews, but it does have a culture of running thematically-themed workshops. A neuroscientist with some ML exposure can go to one of these and get the gist, which is not always the case for the talk tracks at most ML conferences.

I went to the gflownet workshop at Mila last week and returned delighted. Gflownets are something of a Mila specialty, having been introduced in a paper led by Emmanuel Bengio–Yoshua Bengio’s son–in 2021. Powerful but pretty obscure outside of Mila, gflownets have a ton of potential as samplers in otherwise intractable problems. They’ve proved quite powerful in other domains like drug discovery and materials science, and I hope someday soon in neuroscience. Here is my quick intro to these models to get you up to speed.

What’s a gflownet anyway?

Gflownets, short for generative flow network, are a class of algorithms for sampling from a probability distribution. Like rejection sampling, slice sampling, or MCMC, gflownets sample from arbitrary (unnormalized) distributions. They’re particularly well-suited for sampling from objects with compositional structure, like sets, trees, and graphs.

Gflownets differ from classic sampling algorithms in that they’re trained using a supervised loss. Training a gflownet proceeds in the style of online reinforcement learning: you use a variant of an amortized inference network F to generate a sample with the current (bad) policy, evaluate the reward (i.e. the unnormalized likelihood of the sample), and adjust the weights of that network F to get better samples eventually. At convergence, the gflownet can be used to sample very cheaply from the target distribution–amortized inference.

Sampling in a gflownet looks a lot like traversing the state/action space as a reinforcement learning agent. Let’s say you want to sample from a gflownet trained to generate sets of characters of the Latin alphabet. You might start with the empty set ∅. A neural network with a softmax output F(·, ∅) would be used to decide whether to add one of the 26 characters or the stop character □, i.e. using a multinomial sample. Let’s say you sample the letter M. Then you would evaluate the network again, this time on F(·, {M}); and so on until you get a set of letters from 0 to 26 characters long.

Let’s talk about the name: generative flow networks. One key property of the sampler, once trained, is that the amount of probability flowing through one of the intermediate states of the sampler is the same as the outgoing probability. This is like (physical flow) in a river network: water in, water out. Thus, flow. The net in gflownets does not refer to a neural net. It refers instead to the Markov decision process (MDP) which leads to a sample, itself a network. Calling them gMDPs or just generative flows would be a little less confusing IMO. Naming is hard.

Grokking gflownets

As you can see, gflownets mix concepts from two different fields: sampling and reinforcement learning. It’s not that the concepts are really hard to grasp in and of themselves, it’s that gflownets arrange them in unusual ways which can make them hard to grok. You need to sit down with these ideas and let them simmer for a little bit to put them all together. The workshop did a really good job of presenting the background from the two fields to help bridge the gap.

Another barrier to gflownet adoption is that it can be hard to know all the tricks people use to make them work in practice. There’s a well of esoteric knowledge, and it hasn’t yet been translated into an easy-to-digest form (i.e. software that works out of the box). This will happen in due time.

I don’t recommend getting started with gflownets by reading the original paper, or the follow-up gflownet foundations paper; they’re hard to understand and are partly obsolete. Instead, I recommend streaming Day 2 of the workshop for a nice, self-contained intro to the field. There are two accompanying coding tutorials, one for discrete modelling and the other for continuous modelling. The gflownet website contains many other great resources. Once you’ve gone through those, would highly recommend this awesome list.

Losses

One prime example of a footgun in gflownets is choosing the right loss. The loss that is optimized during training is a consistency loss that says that inflows to a node in the decision network are similar to the outflows. This is basically a credit assignment problem: at the end of the process of sampling, you get a sample of a certain probability, and you’re trying to nudge the network such that you would have taken the right branches in the decision tree to get samples which match the probability distribution. Similar to the Bellman equation in reinforcement learning, there are many ways of writing down consistency losses.

You’re going to be tempted to read the first gflownet paper–or maybe the second!–and implement that, and that’s a bad idea. The three most frequently discussed losses are:

• Flow loss. This is a (log) version of the temporal difference (TD) algorithm. It was originally introduced in the first gflownet paper (Bengio et al 2021a), but it doesn’t converge very well in practice, so it’s not really used anymore.

• Detailed balance loss. Some of you familiar with MCMC will recognize this term. In this case, we parametrize both a forward function F_F and its converse backward function F_B such that probabilities match when traversing the tree forward and backward. It was introduced in the gflownet Foundations paper (Bengio et al. 2021b).

• Trajectory balance. This algorithm, introduced in Malkin et al. (2022), does credit assignment on an entire trajectory. That’s the one that is currently recommended as a starting point.

This is the trajectory loss as defined in Malkin et al. (2022):

Here, P_F and P_B are forward and backward flow functions, and Z is the partition function, which is a trainable parameter. With that in hand, training a gflownet is pretty straightforward:

Extensions

• You can condition your gflownets, similar to how you would condition a diffusion model

• You can train a gflownet with observed samples rather than through an environment. This is called MLE-GFN.

• Gflownets work naturally with discrete objects (sets, graphs, trees, etc), but you can extend them to continuous objects. This is important in practice, because many real problems need both discrete and continuous parameters. The math and the implementation difficulties really ramp up in continuous space. This is still a very active area of research.

Software

There are ~3 implementations of gflownet out there, all of which originated around Mila. None of these has yet reached escape velocity but together they should cover a range of use cases.

• torchgfn. Salem Lahlou, et al. This is a low-level library with all the primitives necessary to implement discrete and continuous environments, as well as flow balance, detailed balance, trajectory balance and subtrajectory balance. It contains clean, reference implementations.

• alexhernandezgarcia/gflownet. Alex Hernandez-Garcia. This one is aimed at a higher level of abstraction with more complex environments, so a good place to start for an applied scientist. It has some nice facilities for logging (hydra, wandb). Probably the most active repo in terms of commits, and used internally for several papers.

• recursionpharma/gflownet. Emmanuel Bengio. This one is specialized for pharma and drug discovery applications.

Potential applications in neuroscience

Gflownets allow you to do something important and in general intractably difficult: sampling from funky distributions. You need sampling when uncertainty is an object of interest. Lots of problems can be solved by a good sampler: generating samples, estimating the parameters of a model, estimating the evidence for Bayesian model comparison, and estimating the uncertainty of a learned structure. To be clear, I haven’t seen gflownets applied in neuroscience yet, but it’s something that people are highly interested in.

It turns out that neuroscience is filled with problems involving sampling distributions defined on weird objects. A classic example is BARS, a method to fit splines often applied to peri-stimulus time histograms (PSTHs). Here, the goal is to fit a PSTH with cubic splines, for example, to estimate the latency of a neuron. Splines are defined by the set of their knots (discrete objects taking on continuous location values), as well as their weights in a Poisson regression. This gnarly problem was tackled with reversible-jump MCMC, but it could potentially be tackled with gflownets, where the action space would start from just a constant, adding knots until the entire spline is made.

Another more involved example: learning to generate neurons with intricate anatomy. Neurons are naturally represented as discrete graphs embedded in 3D Euclidian space, and can be sampled one branch at a time, starting from the soma. Note that currently, gflownets with action spaces with >100 actions are considered quite hard to sample from, so one would need to limit the complexity of the space in some way.

Similarly, one could use gflownets to fit and sample connectomes. Here, the action space would correspond to adding connections between nodes in a network, e.g. simulated by leaky integrate-and-fire (LIF) units, biophysically-detailed neurons or mean-field neural masses.

You could classify all these applications under the umbrella of neural data science. I think the vision of gflownets is quite a bit larger: they’re motivated, in part, by modelling cognition, with a Bayesian brain/neuro-symbolic/world models flavour. This gets into highly speculative territory, e.g. here’s a paragraph from a tutorial on gflownets from last year:

Brain sciences show that conscious reasoning involves a sequential process of thought formation, where at each step a competition takes place among possible thought contents (and relevant parts of the brain with expertise on that content), and each thought involves very few symbolic elements (a handful). This is the heart of the Global Workspace Theory (GWT), initiated by Baars (1993,1997) and extended (among others) by Dehaene et al (2011, 2017, 2020) as well as through Graziano's Attention Schema Theory (2011, 2013, 2017). In addition, it is plausible (and supported by works on the link between Bayesian reasoning and neuroscience) that each such step is stochastic. This suggests that something like a GFlowNet could learn the internal policy that selects that sequence of thoughts. In addition, the work on GFlowNet as amortized inference learners [4,5,7,9 below], both from a Bayesian [7] and variational inference [9] perspectives, would make this kind of computation very useful to achieve probabilistic inference in the brain. GFlowNets can learn a type of System 1 inference machine (corresponding to the amortized approximate inference Q in variational inference) that is trained to probabilistically select answers to questions of a particular kind so as to be consistent with System 2 modular knowledge (the “world model” P in variational inference). That System 1 inference machinery Q is also crucial to help train the model P (as shown in several of these papers). Unlike typical neural networks, the inference machinery does not need to be trained only from external (real) data, it can take advantage of internally generated (hallucinated) pseudo-data in order to make Q consistent with P.

Boy, that escalated quickly! From my vantage point as a humble non-Turing-award-winner, it seems like another attempt at a Bayesian brain theory with a different amortized inference scheme than the usual variational inference (VI) based one (i.e. FEP). My understanding is that this is still highly conceptual rather than a concrete proposal, but I’m sure this will evolve rapidly.

TL;DR

1. GFlowNets sample sequentially from unnormalized probability distributions defined on sets and graphs

2. GFlowNets mix concepts from MCMC and RL

3. The best sources of information for GFlowNets are workshops and tutorials; read the papers after

4. They’ve been used for drug discovery and materials science, not yet for neuroscience, but they have a lot of potential.

This paper in Neuron showcases some very strong results in modelling the retina and mechanistic interpretability. They fit CNNs to retinal ganglion cells (RGCs) of retinas responding to natural scenes. RGCs are the final stage of processing in the retina; they’re downstream from the rods & cones as well as horizontal and bipolar cells. It turns out that if you just use the outputs of the RGCs to train your network, you get the intermediate units for free: the units in the intermediate layers of the CNNs map nicely to real interneurons in the retina. That means you can use all sorts of mechanistic interpretability tricks to probe the intermediate layers of the model, with confidence that these units map more or less 1:1 to real neurons in the retina.

That’s a very strong form of alignment between artificial and biological neural networks (see this blog post for an overview of the different degrees of alignment between brains and ANNs and what they can be used for). They use this to unravel different circuits in the retina responsible for different nonlinear phenomena fastidiously documented by physiologists over the years. I think if we can bring this level of detail from the sensory periphery to studies of the cortex, we’re in for exciting years ahead.

Robustified ANNs Reveal Wormholes Between Human Category Percepts

In this epically named paper, the authors find adversarial examples for humans. A little bit of background: some of the best neuroAI models of the visual system are adversarially robust, that is, not sensitive to small image perturbations. This seems to align better with human perception than vanilla networks. Still, adversarially robust networks have adversarial examples, just at larger scales (larger epsilon). In this work, they ask a related question, which is whether the adversarial examples of adversarially robust networks also fool humans. And they do! It turns out that the adversarial examples of adversarially robust network perform small image manipulations which can fool humans to switch to a different semantic category (hence, wormholes between categories). Previously, this kind of effect was demonstrated anecdotally using image manipulation (i.e. Photoshop), and it wasn’t clear how far one could push the idea.

I think this is an exciting avenue because it points towards making networks more robust using a careful empirical approach. We can translate the conventional saying “humans are robust to adversarial examples” to a more mathematically precise “humans’ category boundaries are smooth in epsilon balls around examples” to “humans’ category boundaries are mostly smooth in epsilon balls around examples except along dimensions of high curvature corresponding to situation x, y and z”. What are these situations x, y and z? Are humans truly invariant or robust in these situations, or is it an illusion caused by not looking closely enough? The manipulations demonstrated tend to be highly localized, and so perhaps a more refined notion of epsilon-ball will take into account foreground-background distinctions.

End-to-end topographic networks as models of cortical map formation and human visual behaviour: moving beyond convolutions

There is no accepted mechanism by which weight sharing can be implemented in cortex1. That means that the visual cortex cannot literally be a CNN, and therefore must derive its relative spatial stationarity from the statistics of the natural world and genetic and development biases. What the cortex is, instead, is locally connected: neurons lie on a 2d sheet, receive local input, and project to other areas locally. A complete model of cortical visual processing should swap convolutional layers for local layers. It turns out, however, that it’s not that straightforward to implement a locally connected neural network in an efficient way, and most models thus far have only used a single topographic (i.e. locally connected layer), frequently to model V4 → IT.

This paper does it! Swap out the convolutional layers for locally connected layers, train on categorization on ecoset, an ecologically motivated alternative to ImageNet. I was really curious how they managed this feat, because I went down this exact path last year, even going to the extent of dabbling in Triton to compile custom locally connected kernels for PyTorch. The answer, in the methods, sent me into a rage: turns out there’s a function that does this directly in Tensorflow. ¯\_(ツ)_/¯.

They found some really nice topographic maps that look a little like orientation pinwheels in V1. They also found some evidence (pretty weak in absolute terms) for a better match to human visual biases. Now, once we get into models which have non-stationary properties over space and that approximate foveation, I think we have to think a bit more deeply about the visual diet fed into the model. Images in ecoset, on average, have objects centered within images, because people generally point their cameras at things of interest; but it’s not quite the same as saying that what is fed is an approximation of the foveated diet a human gets. I think it’s possible to approximate that diet by cropping images in proportion to the probability of foveating on objects under the crop, using estimates of salience from eye- and mouse-tracking.

Bispectral neural networks

I heard about this paper from the TWIML podcast, where the first author, Sophia Sanborn did a fantastic job covering a ton of ground in 45 minutes: analytical philosophy, Hubel & Wiesel, sparse coding, group theory, geometric deep learning, and finally, this paper. In Olshausen & Field’s classic work (1996), they found that the responses of neurons in V1 resembled a sparse code for natural images. Hence, there’s a convergence in representation between an optimality principle and the brain.

The work here was motivated by a similar optimality principle, equivariance to identity-preserving transformations. If you translate an image, its power spectrum is preserved; in addition, its phase spectrum maintains a definite set of relationships (i.e. you get a linear phase advance with frequency when you translate). So the Fourier basis is a very good basis for capturing translations. But what about other groups? It turns out that there is a preserved quantity that gets preserved in the Fourier basis under translation, which is the bispectrum. What the authors show is that you can learn the equivalent of Fourier bases on arbitrary groups by imposing that the bispectrum is preserved.

What does that have to do with brains? I think we can connect the dots from Sophia Sanborn’s description of her thesis (which I haven’t been able to find online):

A core hypothesis that I advance in my PhD thesis “A Group Theoretic Framework for Neural Computation” (UC Berkeley, 2021) is that the brain evolved to efficiently encode this transformation structure through the use of group-equivariant representations.

I’m excited to peek more into this field by attending this year’s NeurReps workshop and watching last year’s.

Emergence of emotion selectivity in a deep neural network trained to recognize visual objects

The authors take a look inside a CNN trained on Imagenet and find that there are some units which respond selectively to the valence of a battery of images: neutral, pleasant and unpleasant. They find that ablating these units makes it harder to infer the valence of other items, hence finding a causal role for the units. Thus, perhaps the intrinsic valence of objects can be inferred without needing hardcoding by 1) projecting onto a high-dimensional object space, as might be done by a CNN trained with supervision or self-supervision and 2) downprojecting to some low-dimensional space. The authors conclude that

these results support the idea that the visual system may have the innate ability to represent the affective significance of visual input

That’s an interesting idea. However, I would venture that at least some affective visual biases are baked in; for example, the intrinsic fear of snakes or the attention-grabbing appearance of faces. So while regular visual features from objects may be informative enough to make an affective judgement, where does the mapping from object space to valence come from?

Bonus: The neural code for “face cells” is not face-specific & Medial temporal cortex supports compositional visual inferences

Right as I was about to fire off this newsletter, two papers showed up on my feed. The first is from Marge Livingstone on face cells. It’s a lovely study of the idea that there’s no such thing as pure face cells: these are generic feature detectors that happen to be very good at faces, and they could respond equally well to roundish things with appropriately located holes, e.g. buttons and electrical outlets. They show that you can predict the selectivity of neurons to faces by mapping out their responses to non-faces. Neat!

The second paper from Tyler Bonnen in Dan Yamins’ lab examines the role of the medial temporal cortex in visual perception (MTC), e.g. perirhinal cortex, which feeds into the hippocampus and closely related structures critically important in memory. They cook up visual tasks which are hard for time-limited humans and easy for time-unconstrained humans; they show that DNNs look a lot like time-limited humans; and they show that time-unconstrained humans with MTC damage are a lot like time-limited humans. All in all, this is evidence that there’s something about the MTC that allows higher-level reasoning across saccades to occur. Definitely the start of an interesting research programme.

I was super excited to see this paper roll by from folks at Mila (Eric, Yoshua, Xu) and fellow Neuromatchers (Grace & Megan). It’s 88 pages long, but it’s quite approachable and I was able to finish in a couple of sessions.

The authors present what is perhaps a controversial view of consciousness & AI: that it can be assessed, that you can make a grid to score current AI systems based on theories of consciousness, and that current AI systems don’t check all the boxes but all the building blocks exist today.

The authors really hone in on the soft problem of consciousness: how do you replicate the computational and operational aspects of consciousness? If you replicate all of these aspects, then if you believe in computational functionalism, you solve the “hard problem of consciousness”: why does it feel like anything to be? Computational functionalism is like duck typing for consciousness: if it walks like consciousness, and talks like consciousness, it’s consciousness. It doesn’t matter whether it’s in silico or in vivo. I highly recommend ‘s recent piece to contextualize these conceptions of consciousness.

What follows is then a grand tour of mechanistic theories of consciousness which are compatible with computational functionalism—global workspace theory, higher-order theory, attention schema theory, etc. Notably, this excludes Integrated Information Theory (IIT), which is substrate-dependent. From these theories, they establish a grid of indicator properties: if you hit all these criteria, your confidence that an agent has the working properties of consciousness increases. To be clear, however, the criteria listed are neither said to be necessary nor sufficient: they’re more like a laundry list of commonly assumed properties about consciousness. Some of the common themes include that:

A conscious agent is embodied, has recurrent processing, attention, a global workspace and higher-order processing. It uses sparse, disentangled representations to build models of the world.

Each of these factors can be instantiated in systems that exist today, but they argue that all the pieces haven’t been put together. I wasn’t entirely convinced by the argument that the pieces haven’t all been put together. They argue, in particular, that a reinforcement learning agent built with transformers doesn’t check all these criteria. But interestingly, the thing that disqualifies is slightly outside of the grid itself:

So the system arguably imitates planning and using visuomotor control to execute plans, as opposed to actually doing these things. (emphasis mine)

That seems to me a distinction grounded in the centrality of amortized inference in conscious agents, which is outside the grid. Earlier, they also argue that transformers can’t hit all the criteria, because they’re not recurrent. You could argue that transformers are a superset of finite-horizon, unrolled recurrent nets; disallowing them to be lumped in with recurrent processing seems to run counter to computational functionalism (see, e.g. Doerig et al. 2019). In other words, while I’m convinced that it doesn’t feel like anything to be PaLM-E, the core reasons to make that statement lie outside of the grid they propose.

Nevertheless, a thought-provoking read, and a good companion and counter-weight to Erik Hoel‘s recent book dealing with a theory of consciousness not compatible with computational functionalism, Integrated Information Theory (IIT).

Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning

The last few years have seen a rise of methods that find dense features in large-scale datasets, and then use these features for transfer learning, whether it’s vision transformers or large language models. This is in contradistinction to the classic methods of sparse feature learning: sparse coding, dictionary learning, and non-negative matrix factorization. The authors find a fairly general result on the identifiability of sparse hidden factors in deep models in multi-task and meta-learning settings. They propose a straightforward dual-loop algorithm to learn latent features and their sparse loadings and show an advantage of these sparse representations in terms of sample efficiency.

My NeuroAI take: There’s a long history of sparse coding as a model of the brain. Neuroscientists made strong claims that sparse codes were, in some sense, better than dense codes. Yet, your average transformer works pretty darn well with a dense representation. What gives? Perhaps sparse representations are more useful in certain regimes, for example when examples are few. It would be interesting to link these results to the Baldwin effect on evolutionary timescales: perhaps there is evolutionary pressure towards sparsity because it can make learning more efficient.

MyoSuite

Not a paper but a piece of software that’s highly relevant to this crowd. Guillaume Durandau, who became a prof in mechanical engineering at McGill this February, came to showcase MyoSuite during the Mila NeuroAI reading group. MyoSuite is a very slick software suite capable of simulating muscles and skeletons of the forearm as well as legs in MuJoCo. He made an impassioned pitch that this kind of simulation software could serve as a basis for a new form of translational NeuroAI: with good models of sensorimotor cortex, the spinal cord, and skeletomuscular mechanics, we could build new therapies for those affected by disorders of movement. The key idea is to use these models to work our way backwards to find optimal stimulation patterns. It’s a very exciting research programme.

MyoSuite is hosting the MyoChallenge at this year’s NeurIPS, where you learn to control a simulated human skeleton for fine forearm movement or locomotion. 20k$+ in prizes! There’s a month until the deadline, so get in there!

Contrasting action and posture coding with hierarchical deep neural network models of proprioception

The MyoSuite talk reminded me of this paper I had put in my stack for a while from the Alexander Mathis lab looking at how the brain learns representations in somatosensory cortex. Using a model of spindles and a skeletomuscular model of the arm, they generated action sequences corresponding to letters. They then trained a neural network on either recognizing the letter that was traced or decoding the trajectory. They found that letter recognition leads to representations that look more like those you would find in single neurons in somatosensory cortex.

My first thought reading this is that this is incredibly cool work: simulations! NeuroAI! Systems that people haven’t looked at from this lens! My second thought was: wait, what? I’m sympathetic to the idea that you can use efference copy to learn good representations in the brain, but this feels like a bit of a stretch. One thing that feels fishy is that you can have fine motor skills without much centralized planning; e.g. think of the octopus. Indeed, the reviews published in eLife, including one from Niko Kriegeskorte, seem to converge on these points: cool idea, great execution, I don’t buy the conclusion. Definitely an interesting new research programme.

Rastermap: a discovery method for neural population recordings

From Carsen Stringer and Marius Patchitariu (friends from the NMA year 1 days) comes this cool method to sort spiking activity in massively parallel recordings like Neuropixels. You can view it as a combination of k-means and a (semi-)exhaustive search to minimize a mixed local and global loss on groups. It does a really good job on real spiking data, but perhaps most intriguingly, it appears to work very well on ANN activity, as illustrated by this analysis of the population activity in RL networks trained on Atari games. One more tool in the belt of mechanistic interpretability?

Send me your paper!

They say that you’re the average of your 5 closest friends; I certainly had the impression this week that my readings were heavily weighted towards papers 1) authored by people at Mila and 2) from people I met through Neuromatch Academy. If you’re outside of that bubble, and you think your paper should be on my radar, comment or send me an email!