Skip to content

Visualizing memory states in transformers

PresentationAyse Demir
researcher
Download →

In transformers, memory is actively retained. Each token is generated by conditioning on prior context, and the model's active state is shaped by what it holds from the past. This talk focuses on making transformers' memory states visible and interpretable. The first part presents a visual vocabulary for retention — attention across layers, attention-entropy landscapes, KV-cache retention fields, and memory-injection overlays — that renders the model's holding of the past perceptible. The second part turns from visualization to applied demonstration. For instrumentation, I use change-point…

Event: Protocol Symposium 2026, Friday, September 25, 2026

Slides and papers

Shared by the speakers for the symposium. Email addresses have been removed.

Topics

symposium-2026protocols

Related resources

Talk

A Common Protocol for Composable, Modular Organizations

Simone Cicero of Boundaryless presents O2A (Open Organisation Alliance), a proposed open standard for making organizations 'composable' by replacing implicit coordination mechanisms like hierarchy and meetings with an explicit, machine-readable contract graph of units, offerings, and relationships. He argues the protocol's compact grammar—three economic primitives (purchase, revenue split, investment) plus composition mechanisms—can represent bespoke institutional arrangements like Haier's Rendanheyi model or joint ventures as 'sentences' in a common organizational language, becoming essential as AI agents enter firms and transaction costs collapse.

Simone Cicero

Talk

Closing Session

Venkatesh Rao delivers a closing synthesis of the five-day symposium, highlighting recurring themes like 'new nature' (Sean Stevenson's test: a system is new nature when there's nobody to appeal to), the interiority/exteriority tension in AI discourse, and the balance between playable protocols and grinding enterprise IT work. Timber Stinson-Schroff co-hosts a town hall where attendees share standout moments, Rao announces the Protocol Institute's transition to fiscal sponsorship under David Lang's Experiment Foundation following the end of Ethereum Foundation funding, and the group plugs the Monstrous Times fiction magazine spinoff led by Sachin Benny and Spencer Nitkey plus an upcoming November 'book writing month.'</summary>

Venkatesh Rao, Timber Stinson-Schroff

Presentation

Continuity Infrastructure

Continuity is related to memory, but it is something else: a substantial relationship between the past and present of some thing, greater than its persistence or its identity. A rite can persist in texts long after anyone performs it. It is continued only when it is taken up again and its significance to the present is renewed. Continuity in this sense is never simply given. It has to be instituted, against a natural drift towards fragmentation. This is most explicit where there has been a temporal break: in the revival of a language, a culture or a national identity, where a present must…

Ruthvik Peddawandla