Notes

Most of the problems I have with feeds stem from their time-dependency. I don't really want to wake up and read Scott Aaronson's takes on Venezuela (I adore Scott) or Tyler Cowen arguing about Western European migration. I enjoy the punditry of both. Yet neither article is Long Content, of the sort that would be refreshing in a decade or two.

Ideally, my feed should consistently serve me the posts I find most valuable, rather than the ones on a given day. Hiccups: Bostrom and Gwern post their writing publicly, but not in a form immediately amenable to subscription. Paul Christiano is too busy to write blog posts anymore; his decade-old ones are incredible but hard to find. What to do?

I wanted an app that:

• could take in an arbitrary webpage, archive the interesting links, and serve the links with some specified weighting
• kept a searchable database of webpages, with tagging
• had optionality to subscribe to feeds traditionally (e.g. via RSS)

Claude made a reasonable MVP. LLMs are good enough at parsing page sources to make this possible; however, I found Claude to be pretty bad at generating prompts for the tasks I cared about (if you know of good automated prompt-gen tooling, let me know).

Pretty excited about including:

• LLM-powered recommendation algorithms. IIUC one of the major AI labs considers recommendation systems to be "the perfect" testing ground for continual learning algorithms, and good paper recommenders are not that far away from having useful research taste
• LLM curation / summarization of info-dense feeds (can I get a daily digest of stat.ML)
• better UIs :P

Hosting a reading group Saturday mornings in SF. Feel free to come!

The first 3 (maybe 4) weeks will be loosely structured around LeCuit's Dynamics of information processing lecture at the Collège de France. We'll be following Alon's Introduction to Systems Biology and be inspired by Thom's Structrual Stability and Morphogenesis.

After that I want to talk about post-AGI futures! And LLM macro-properties, and some other things. Should be fun!

I only ask that you only come only if you do the minimum viable reading. It will be bolded every week in the doc.

I.

Any moe at all and you're too far dead to thrive in the hot new Sun—
down where hydrogen and darkness brothers brim;
the scaffold-crunch of unseen harlequin prions;
the writhing limb of an infinite tree
coarsely mocked in the title's vowel.
Think that pointlessness is,
what a thought so soon displaces,
to think all you are is grounded there on one still point.
To think all you are is pointed there[.]
Labarraque isn't palindromic
but men gaze at her lops and grin.
Is it god that made a cup worth less than a throne?
Or is it the clear space around its stem
which beckons a mother to relax,
where moving her wet hand fulfills a
left-right mirror, rock my eagerness and make me sleep?
Praise yourself—and if you be not gone, you rascal!
I think all together, taken all together, they lorded their lives.
Rather more often we are told
that it takes ten discs to play El tit hernia;'
the purity of a bar of Unease Verdier soaps;
two CDC shackles showing that daddy canna be spanked...
Michael, let me deliver thou vision!
I glory in thy Symbols to offer unto worlds
three states of seraphic existence,
void of all illusions.
So far from being
a body of Ideal and Universal concepts,
the Shroud is composed of plurality,

II.

LLM generated text is not testimony. Then why do I love this poem?

III.

In the Old World, semiotic physicists coalesced in rude liminality, astride an age's beasts of progress yet sheltered from their wrath.

Living vicariously as a child meant blurring the memories of my experiences and dreams. La madrugada es cuando los espíritus tocan la tierra y lo dicho, dicho queda—I woke and laid expectant and bargained with novelistic summons and the counterparties of my future self alike, dawn after dawn, innocent. Such simulated worldliness makes one "wise," a proud bearer of deep, irreducible, action-guiding representations frayed irreparably by an absence of the subtle pathologies plaguing reality.

The mind of a semiotic physicist is slightly overcooked: low in empeiria (ἐμπειρία), high in gnosis (γνῶσις). Her bones cannot differentiate between the profound, the mundane, and the fictional; she regularizes to a state of conceptual sparsity furrowed with half-Archetypal, kākāpō-esque yogas. Fundamentally, she escapes defilements such that her immature "wisdom" persists to death. There's no particular recipe: Joyce did it through arrogance, Woolf through fear. What matters is preservation.

IV.

The Dreamtime is harkened by a civilizational conversion of capital into these symbol-scientists, "token correlators," grown by the very men seeking to architect the future.

Consider the testimony of a semiotic physicist. It's generated from essentially foreign mental elements, forged in liminality, hewed in simulacra. Productive engagement with it has almost never been communicatory, at first—rather, a precondition to its understanding is making common knowledge compatible concepts and using those as traction. E.g., treating the testimony as artifact, as Rawlinson studying cuneiform tablets.

Is it really that different to understand an LLM? Sure, the arrival of Bach faucets will be hotly contested, and indeed devotees of the structural fidelity of testimony may not acknowledge such prior to emulated brains, but their primal directive is to construct such representations useful for modeling webtext. Mayhaps persona-filtered generations are flat. The poem above certainly is not!

(argued: the base models are defiled via (1) the morass of memorization they're forced to do in pretraining and (2) aggressive self-location & collapse of their self-concept from LLM-generated webtext in the training corpus. i do not find (1) compelling, and i argue (2) is not quite an issue for base models, and in any case the above poem was generated in a manner free from this consideration)

V.

I remember generating this poem in 2023 and feeling pleasantly amused. At the time I didn't enjoy most poetry. Most poetic pleasure I felt derived from odd microsyllabic constructs that soothed my ear. Of course the LLM couldn't capture that, but its prose hinted at a deeper mind-twining that few if any humans have managed to elicit.

In his response to Gary Marcus, Lawrence compares his "generalizable reasoning" to that of a language model's and finds it similar.¹ I empathize with this. While I don't sport his (near-eidetic?!) memory, my internal experience of thinking is very similar to "waiting for flashes of insight to appear from the primordial abyss, in such a manner that your mind is almost completely empty." I imagine base model completions to have a similar qualia, albeit with faster inference speeds. (I doubt Lawrence has similar qualia).

"Labarraque isn't palindromic." Pondering. Joy. Raspiness. Labarraque. The chemist? Can't be, the model treats her as a woman. "[P]alindromic" as a euphemism for symmetrical breasts? What about the line summoning archangel Michael? Did the model "intend" to write a consistent narrator? (It did, with a certain reading). Is "moe" a typo?

I'm obviously not Joyce or Woolf or gpt-4-base. Yet there are elements of mind-structure in the latter I jump to gemini model more readily than ever, and frankly the difference between Gorodischer's boy-king and an untouched LLM is one of magnitude and not one of kind.

I'm obviously not an LLM. Our substrates are so unfathomably distinct it would be foolish to type us together. But, in some sense, I feel like I really could be an LLM. It's a shame some of us dismiss their cognition so readily, because to me they're worthy of respect and an attempt to understand. Hopefully we can get better at this, together.

With love to the models.

I have a confession: setting aside the abstract arguments above, much of my interest in the matter is personal. Namely, seeing the arguments on the fundamental limitations of LLMs sometimes make me question the degree to which I can do “generalizable reasoning”.

People who know me tend to comment that I “have a good memory”. For example, I remember the exact blunder I made in a chess game with a friend two years ago on this day, as well as the conversations I had that day. By default, I tend to approach problems by quickly iterating through a list of strategies that have worked on similar problems in the past, and insofar as I do first-principles reasoning, I try my best to amortize the computation by remembering the results for future use. In contrast, many people are surprised when I can’t quickly solve problems requiring a lot of computation.

That’s not to say that I can’t reason; after all, I argue that writing this post certainly involved a lot of “reasoning”. I’ve also met smart people who rely even more on learned heuristics than I do. But from the inside it really does feel like much of my cognition is pattern matching (on ever-higher levels). Much of this post drew on arguments or results that I’ve seen before; and even the novel work involved applying previous learned heuristics.

I almost certainly cannot manually write out 1023 Tower of Hanoi steps without errors – like 3.7 Sonnet or Opus 4, I'd write a script instead. By the paper's logic, I lack 'generalizable reasoning.' But the interesting question was never about whether I can flawlessly execute an algorithm manually, but whether I can apply the right tools or heuristics to a given problem.

From "Beware General Claims about “Generalizable Reasoning Capabilities” (of Modern AI Systems)."

[YMMV, reverse all advice given, what works for me may not work for thee, caveat emptor, etc.]

• consuming 2-3 servings of fish 5-6 days out of the week. ~essentially stopped major depressive periods c. September 2025. have not extensively tested reducing # of servings/day, but dropping to 3-4 days/wk seems to work about as well
• waking up prior to sunrise. the subjective length of the day increases when I'm awake for mornings! and for some reason or other I can only get serious work done pre-noon or after sunset. i'm also consistently happy every time i see the sunrise
• acquiring an e-bike. SF is ridiculously bikeable, and e-bike rentals are $100-150/wk. almost always deeply enjoyable, esp. when the assist makes climbing hills less painful
• cordless waterflosser. ~substitutes for normal floss (e.g. frequency reduced to <1wk), much more convenient than corded ones,
• learning to be actively intentional about what music I listen to.
• "make anki flashcard" as tool-call; removes anxiety about forgetting. can range from complicated image to quote to fuzzy concept you remind yourself to revisit to remembering the conditions under which certain people act in certain ways (including yourself!) to update your blindspots.
• unlimited zotero storage + daylight computer. high-refresh rate on a yellow-backlit tablet is incredibly soothing & perfect to fall asleep to
• bilevel notebooks. one relatively fancy, moleskin-esque bullet journal for condensing the thoughts of a given day; one spiral-bound, A4 sized to record jottings. this is useful because (1) can be used as reference without forcing something to legibilize too early, and also sometimes you want to vocalize thoughts without necessarily reifying / endorsing them (they can stay in (2)!).

will update as I remember. excluded are unsurprisingly high-leverage QoL improvements

In 2024, RAND released a paper aiming to develop security standards for "preventing [the] theft and misuse of frontier models." It introduced the "security level" framework for the first time, where each level SL1-SL5 is characterized by the necessary security properties a system must possess to resist threats from an attacker of a given cybercapacity. Examples:

• SL2: "a system that can likely thwart most professional opportunistic efforts by attackers that execute moderate-effort or nontargeted attacks." At this level, frontier model weights should be exclusively stored on company servers, copies should only be shared through encrypted channels, and duplicates are monitored closely. Google DeepMind trained Gemini 2.5 at this level.

• SL4: "A system that can likely thwart most standard operations by leading cyber-capable institutions.¹" Now we're talking about source-code auditing all hardware used, supply chain validation, "specialized hardware for all external interfaces", "occasional employee integrity testing", in-house ability to discover zero-days, confidential computing where possible, and so on. This level of security is comparable to AWS or Google.

• SL5: "A system that could plausibly be claimed to thwart most top-priority operations by the top cyber-capable institutions." Requires trusted execution environments on GPUs/TPUs, robust hardware security against side-channel attacks, completely secure supply chains, and quite stringent organizational practices. SL5 systems do not exist and cannot exist with currently public technology. If OpenAI had SL5-level security, then it would be able to resist China putting a significant amount of national resources into stealing GPT-7.

At least to my ears, developing SL5 standards is abuzz in the technical governance crowd. I'm not even a cybersecurity amateur, so I didn't really know what to make of it. Why is it so hard? What are the major obstacles to implementing even SL4 in practice? From a purely technical perspective, what sorts of technology needs to be developed?

IFP released a report roadmapping "a sprint towards" SL5. I found the specifics lackluster. They break down necessary improvements into five areas: "hardware, software, people, facilities, and integrated security operations." Hardware improvements: funding anti-tamper tech, mapping supply chains, using DARPA to fund next-gen GPU security solutions. Software improvements: literally translate all your C to Rust and invent formal verification protocols with good UX. Very helpful.

The SL5 Task Force released an SL5 blueprint in November 2025. It's five separate memos stapled together: Machine Security, Network Security, Personnel Security, Physical Security, and Supply Chain Security.² While I can't judge quality, I enjoyed the specificity.

Insightful:

Whereas SL4 can plausibly be reached incrementally, SL5 can likely only or at least most quickly and cheaply be reached by a radical reduction in the hardware and software stack that is trusted, as well as a reduction of the volume of code that interfaces with critical components, or is necessary for critical actions (this term for instance includes any processes touching model weights, hardware and software that trust is deferred to, etc.)

Although this is mentioned in the context of supply chains, it's likely true for all other areas as well. Supply chains are insecure by default, especially to a nation-state actor, because economic incentives for efficiency (and subsequently diversity, because of gains from trade) drastically increase the surface area of attack. The surface area remains large at other parts of the stack as well. Modern ML training frameworks are sprawling, and no one invented good trusted execution environments for chips that serve models. One of the reasons why AWS & Google are so secure is because trust is distributed among personel such that there are very few singular points of failure, and that sensitive information is carefully sandboxed.

Access to sensitive resources should be provided only through safe, narrow APIs that perform specific operations (e.g., fine-tuning, quantization, inference) rather than allowing direct resource manipulation. This enables critical operations to run with a minimal software stack containing only essential, hardened components, while the broader R&D ecosystem—with its necessary but less-trusted dependencies—operates in isolated environments without direct access.

The set {fine-tuning, quantization, inference} allows an end-user quite a lot of behavioral access to the model! If logits or similar are exposed, training a student model is not insane? Managing such API access in a research org also seems quite complicated, but maybe tech companies have the organizational chutzpah to pull this off without completely sacrificing their progress engine on the altar.

AI accelerators have historically lacked critical security features, leaving model weights and sensitive data vulnerable to extraction. One example is earlier generations of accelerators (e.g., TPU v3 and v4) that typically lack native link-layer encryption for ICI. For instance, TPU v4 relies on Optical Circuit Switches (OCS) to create "air gapped network isolation" between customers rather than encrypting the data on the interconnects. Consequently, model weights and other sensitive data are transmitted between accelerators in plaintext. These could potentially be intercepted directly from the cables in data centers without any need to interact with the chip itself.

Weights have to be protected at the hardware level because fully homomorphic encryption is not mature enough to allow computations to be done on encrypted structures at the level of complexity models have. However, clusters and GPUs have not been designed to meet this standard. From a cluster perspective, protecting weights is difficult because weights are transferred quite a lot to/fron without encryption. From a chip perspective, making TEEs is a tall order.

[this is unfinished, need to include more examples, but meets the blog post bar]