How do LLMs count with “thinking” mode on or off?
Published:
Retrieval strategies, count representations, causal interventions, and training dynamics in long-context counting.
Posts by Tags
LLM
LLM Optimizers: From SGD and AdamW to SOAP, Muon, and Scalable Training
Published:
Implementation-first notes on optimizer geometry, AdamW, memory and distributed systems, and matrix-aware methods including SOAP and Muon.
Tokenization: From Building BPE from Scratch to Language Models Without a Fixed Tokenizer
Published:
Implementation notes on byte-level BPE, fixed-tokenization failure modes, and dynamic byte-level models.
Life
Shallow ideas on the future of Statistics.
Published:
I recently posted a paper whose theoretical derivations got an enormous amount of help from GPT (even though 5.6 Sol wasn’t out yet at the time). My senior labmate and I took the proof outline GPT gave us, refined it, and distilled the insights. Over the summer, I talked with a lot of people, and one question kept coming up: what is the point of theory, and where is its future?
Welcome to my personal website!
Published:
Today is 2026.3.23, I finally have my personal homepage with the help of Codex!
RL
Can One LLM Win at Everything? Finding the Pareto Frontier
Published:
An interactive introduction to Pareto-frontier identification in multi-objective LLM arenas and adaptive sampling with VB-EGE.