← Thinking practiceClassic papers and essays
The pieces that most of what I work with rests on, most of them short, all of them free to read. Each comes with a summary, the idea worth keeping in one line, and what it changes in how you think, so the list is useful before you have read a word of it.
16 pieces · about 12 hours of reading · 0 read so far
Information and computing
1945·essay·45 min·To read
Vannevar Bush · The Atlantic Monthly, July 1945
Writing as the war ended, the director of the US Office of Scientific Research and Development asks what scientists should turn to next and answers: the record. Human knowledge was growing faster than anyone could consult it, and indexing by alphabet and category fights the way the mind works, which is by association. He imagines the memex, a desk that stores a library on microfilm and lets its owner build and share trails of linked documents. Along the way he predicts dry photography, voice typing, and machines that select by association rather than by index.
The idea to keep
Knowledge is only as useful as the means of finding it, and the mind finds by association, so build tools that link.
What it changesYou read hypertext, the web, personal knowledge tools and retrieval-augmented systems as one seventy-year project, and judge a new tool by whether it helps you make and follow trails rather than by how much it stores.
Models: The adjacent possible, Network effects, Multiple discovery
1948·paper·3 h·To read
Claude Shannon · Bell System Technical Journal, 55 pages in the corrected reprint
Shannon separates the meaning of a message from the problem of getting it across, and then solves the second problem completely. A source is a random process; its entropy, measured in bits, is the average number of yes-or-no questions its output takes to pin down; a channel has a capacity, also in bits per second; and messages can be sent at any rate below capacity with as few errors as you like, and at no rate above it. The paper defines the bit, proves the coding theorems, and introduces the noisy channel, redundancy and the idea that English is about half redundant.
The idea to keep
Information is the resolution of uncertainty, it can be measured in bits, and every channel has a hard capacity that no cleverness can beat.
What it changesYou stop thinking of compression, error correction and transmission as separate tricks and see them as one trade against a fixed budget. It also gives you a precise sense of how much a message, a log line or a model's output actually tells you.
Models: Entropy, Chunking, The map is not the territory
1950·paper·1 h 15 min·To read
Alan Turing · Mind 59(236), pages 433 to 460
Turing replaces the question 'can machines think?' with a test he can state: whether an interrogator, typing to a hidden machine and a hidden person, can tell which is which. He then works through nine objections, theological, mathematical, from consciousness, from disabilities, from Lady Lovelace, and answers each. The last third proposes building a machine that learns like a child rather than programming an adult mind, and guesses that by the end of the century the words would have shifted enough that talk of machines thinking would not be contradicted.
The idea to keep
Replace an unanswerable question about the inside of a system with an observable test of its behaviour, then argue about the test.
What it changesYou notice how often a hard argument is really about a definition, and you reach for the behavioural version of the question. Reading the nine objections in 1950 also shows how little the debate about machine intelligence has moved its shape.
Models: Thought experiments, The map is not the territory, First principles
1978·paper·35 min·To read
Leslie Lamport · Communications of the ACM 21(7), 8 pages
In a system of processes that communicate only by messages, there is no single clock, and 'before' has to be defined. Lamport defines the happened-before relation from two facts, events within one process are ordered and a message is sent before it is received, and shows that logical clocks, counters that tick on each event and jump forward on receipt, respect it. From a total order built on those clocks he derives a distributed mutual exclusion algorithm with no central coordinator, then shows what physical clocks need to keep the order consistent with the one observers would see.
The idea to keep
In a distributed system, order is something you construct from causality, not something you read off a clock.
What it changesYou stop assuming timestamps from different machines can be compared, and you start asking what a system's notion of before actually is. Vector clocks, consensus protocols and most of the CAP discussion read as footnotes to these eight pages.
Models: The CAP theorem, The map is not the territory, Emergence
Software and craft
1968·paper·8 min·To read
Edsger Dijkstra · Letter to the editor, Communications of the ACM 11(3), 2 pages
A two-page letter, retitled by the editor, arguing that the quality of a programmer is inversely proportional to the density of go to statements in their programs. The argument is about understanding, not taste: a reader needs to relate the text of a program to the process it describes, and with structured constructs the position in the text plus a few loop counters tells you where the process is, while an unrestricted jump makes the current state depend on the whole history of the run. The letter is the public start of structured programming.
The idea to keep
Code is read by people tracing an execution in their heads, so a construct is good to the degree that the text tells you what the process has done.
What it changesYou judge any language feature, abstraction or architecture by how much a reader has to hold in their head to follow it, which is the question behind most arguments about goto's descendants: exceptions, callbacks, global state and implicit control flow.
Models: Chunking, Gall's law, The map is not the territory
1984·talk·10 min·To read
Ken Thompson · Turing Award lecture, Communications of the ACM 27(8), 3 pages
In three pages Thompson shows how to plant a back door that survives inspection of every line of source. Modify the C compiler to insert a hole when it compiles the login program; then modify it again to insert both changes whenever it compiles a compiler; then remove the changes from the source. The compiled compiler now reproduces the back door forever, and nothing in any source file shows it. The moral is stated plainly: you cannot trust code you did not totally create yourself, and the problem is not confined to compilers.
The idea to keep
Trust in a system bottoms out in trust of the people and tools that built the tools, not in anything you can read.
What it changesSupply-chain security, reproducible builds and the question 'what does this depend on that I have never looked at' all become live. You also hold a little more humility about what a code review can and cannot show.
Models: Leaky abstractions, Margin of safety, Hyrum's law
1985·paper·45 min·To read
Peter Naur · Microprocessing and Microprogramming 15; reprinted in Computing: A Human Activity (1992)
Naur argues that the real product of programming is not the code or the documentation but a theory, in Gilbert Ryle's sense: the understanding in the programmers' heads of how the program relates to the world it serves and why it is built as it is. He gives two cases, a compiler handed to a new group and a large system whose modifications went wrong, where the text and documents were complete and the new team still could not extend the system sensibly, because the theory had not come with them. Programmer turnover and 'maintenance' as a separate activity look different once the theory, not the text, is the asset.
The idea to keep
A program is the written-down part of an understanding that lives in people, and the understanding is what lets the program be changed well.
What it changesYou treat onboarding, pairing and keeping a team together as engineering decisions rather than HR ones, and you read a codebase whose authors have left as a text whose theory has died. It is also the sharpest frame for what a language model does and does not have when it edits code.
Models: The map is not the territory, Chunking, Brooks's law, Conway's law
1986·paper·45 min·To read
Fred Brooks · University of North Carolina technical report 86-020; also IEEE Computer, April 1987
Brooks divides the difficulty of software into essence, the specification and design of the conceptual structure, and accident, the labour of expressing it in a language on a machine. Past gains, high-level languages, time sharing, unified environments, attacked accidents, and he argues the remaining accidents are too small for any one development to give a tenfold improvement in a decade. The essential difficulties are complexity, conformity, changeability and invisibility. His hopes are buy rather than build, rapid prototyping, growing software incrementally, and great designers.
The idea to keep
Most of the cost of software is in deciding what it should be, and no tool that only speeds up expressing a decision can remove it.
What it changesWhen a new language, framework or model promises to make software easy, you ask whether it touches the essence or an accident. It also explains why the hard part of an AI system is still the specification, and why 'grow it, don't build it' keeps winning.
Models: Amdahl's law, Brooks's law, Gall's law, YAGNI (you aren't gonna need it)
1986·talk·1 h·To read
Richard Hamming · Bell Communications Research colloquium, 7 March 1986 (transcript)
Hamming, who worked beside Shannon at Bell Labs, asks why some scientists do work that matters and most do not, and answers from thirty years of watching. The people who did great work had courage, worked on important problems, kept their office doors open, turned their problems around until they fit the tools they had, and sold their results. He is blunt about the costs: the drive that produces great work is hard on the people around it, and the open door costs you interruptions you must decide are worth it. The talk's refrain is a question: what are the important problems in your field, and why are you not working on them?
The idea to keep
Great work comes from choosing important problems and compounding effort on them, and most people never make the choice deliberately.
What it changesYou audit what you spend your attention on against what matters in your field, and you notice that the open door and the closed door are a real trade rather than a personality.
Models: Compounding, Deliberate practice, Circle of competence, The opportunity cost of time
2009·essay·10 min·To read
Paul Graham · paulgraham.com, July 2009
Managers run their day in one-hour slots, and a meeting is just another slot. People who make things, programmers and writers, work in half-day units, because hard work needs a long run-up and a single meeting in the middle of an afternoon breaks it into two pieces too small to start anything in. Graham argues that most organisations run on the manager's schedule without noticing that it costs their makers most of a day per meeting, and describes the workaround Y Combinator used: office hours at the end of the day, when the damage is least.
The idea to keep
An hour costs a manager an hour and a maker half a day, so meetings should be scheduled by who is paying.
What it changesYou batch interruptions, defend blocks rather than hours, and read a calendar full of scattered thirty-minute meetings as the destruction of a working week rather than as a busy one.
Models: The opportunity cost of time, Bottlenecks and the theory of constraints, Redundancy and slack
2013·essay·25 min·To read
Paul Graham · paulgraham.com, July 2013
Founders expect a good product to spread on its own; Graham argues that almost every startup had to be pushed by hand at the start. Recruit users one at a time, make them so happy they tell others, do things for them by hand that will later be automated, and narrow the first market until you can dominate it. His examples are Airbnb's founders photographing apartments in New York and Stripe installing itself on founders' laptops. The essay reframes the unscalable early work as the way to learn what the product should be, not as a detour from building it.
The idea to keep
Early on, the unscalable work of recruiting and delighting users by hand is the product discovery, and it is where the scalable version comes from.
What it changesYou stop treating manual effort as a sign the product is failing and start treating it as the cheapest experiment, and you look for the small market you can own before the large one you cannot.
Models: Gall's law, Critical mass and tipping points, Network effects
2023·essay·1 h 10 min·To read
Paul Graham · paulgraham.com, July 2023
A long essay that tries to say what the recipe for great work has in common across fields. Choose a field that suits your aptitude and that you are curious about, learn enough to reach the frontier, notice the gaps there, and work on the ones that excite you. Graham argues for following curiosity over planning, for the compounding value of consistent work, for writing to find out what you think, and for the particular kind of hard work that feels like play. He treats ambition, taste, originality and luck in turn, and is frank that the essay is a map rather than a guarantee.
The idea to keep
Get to the frontier of something you are genuinely curious about, then work on the gaps that look like anomalies to you and boring to everyone else.
What it changesYou weight curiosity as evidence about what to work on rather than as a distraction from the plan, and you take the gap between your taste and your output as the thing to close.
Models: Compounding, Deliberate practice, The adjacent possible, Circle of competence
Science and method
1974·talk·15 min·To read
Richard Feynman · Caltech commencement address, printed in Engineering and Science, June 1974
Feynman names the activities that have the form of science without its substance after the Pacific islanders who built runways and bamboo headphones after the war and waited for the planes. The missing ingredient is a kind of scientific integrity: bending over backwards to report everything that could make your result wrong, including the things you did not do and the alternative explanations you could not rule out. He tells the story of Millikan's charge on the electron and the slow drift of later measurements towards the right value, as researchers discarded results that differed too much from Millikan's, and ends with the rule that the first principle is that you must not fool yourself, and you are the easiest person to fool.
The idea to keep
Scientific honesty is actively reporting what would undermine your own result; you are the easiest person to fool, so the discipline has to run against yourself first.
What it changesYou write up the caveats before the result, you distrust a measurement that lands suspiciously near the expected one, and you learn to notice the Millikan drift in your own work: the extra scrutiny you give to the numbers you did not want.
Models: Confirmation bias, The Feynman technique, Correlation is not causation, Steelmanning
Economics and society
1945·paper·40 min·To read
Friedrich Hayek · American Economic Review 35(4), September 1945
Hayek's argument against central planning is not about motives but about knowledge. The facts an economy runs on, which machine is idle, which shipment is late, what a customer will pay, are dispersed among millions of people as knowledge of particular time and place, and they change too fast to be gathered anywhere. Prices are the mechanism that lets people act on knowledge they do not have: a rise in the price of tin tells everyone who uses tin to economise, without anyone knowing why. The price system is a telecommunications network, and the marvel is how little each participant needs to know to act correctly.
The idea to keep
The knowledge a system needs is dispersed and local, and the best coordination mechanisms transmit it compressed, as prices do, rather than trying to centralise it.
What it changesYou design organisations and systems to push decisions to where the information is, you distrust any plan that assumes a centre can know enough, and you read a price, a latency figure or a queue depth as a message about facts nobody holds whole.
Models: Emergence, Supply and demand, The map is not the territory, Conway's law
1968·paper·40 min·To read
Garrett Hardin · Science 162(3859), 13 December 1968
Hardin's parable is a pasture open to all: each herdsman gains the whole benefit of adding an animal and bears only a share of the overgrazing, so each adds another, and the pasture is ruined. He applies the arithmetic to pollution, to the oceans and, most controversially, to population, and argues that some problems have no technical solution, only a change in values or in coercion. His remedy is mutual coercion, mutually agreed upon. The paper is read today alongside Elinor Ostrom's work, which showed that communities often govern their commons well without either privatisation or a central state, and the entry on the mental model carries that correction.
The idea to keep
When the benefit of using a shared resource is private and the cost is shared, individual rationality destroys the resource unless the users can bind each other.
What it changesYou see shared engineering resources, a staging cluster, a build queue, a team's attention, as commons with the same arithmetic, and you look for the rules the users can set themselves before reaching for a central decree.
Models: Tragedy of the commons, Incentives drive behaviour, Feedback loops
Machine learning
2017·paper·45 min·To read
Ashish Vaswani and seven co-authors · Advances in Neural Information Processing Systems 30
The paper proposes the transformer, a sequence model built entirely from attention, with no recurrence and no convolution. Each position attends to every other through scaled dot-product attention, run as several heads in parallel, with position encoded by added sinusoids; the result trains in parallel across a whole sequence rather than token by token. On WMT 2014 translation it reached 28.4 BLEU English to German and 41.8 English to French, better than the previous best models at a fraction of the training cost. The architecture is the one every large language model since has been built on.
The idea to keep
Let every position in a sequence look at every other directly, and the model becomes both better and far cheaper to train at scale, because nothing has to wait for the previous step.
What it changesYou understand why context windows, attention cost and position encoding are the levers they are in every model you use, and why the field's progress since 2017 has been mostly scale and data on top of one design.
Models: Scale effects and the square-cube law, The adjacent possible, Exponential versus linear growth, Amdahl's law
