This is the two hundredth piece published here. That seems like a reasonable moment to write about the writing rather than about the subject.
\nIt forced me to actually understand things. You can hold a fuzzy model of a technology in your head indefinitely and it feels like knowledge. The moment you try to explain it in a paragraph, the fuzziness becomes visible.
\nI have abandoned drafts because I discovered, four hundred words in, that I did not understand the thing well enough to write about it. Every one of those was more educational than the pieces I finished.
\nIt made me check things. Writing \"X is faster than Y\" in public means someone will ask for numbers. Knowing that in advance changes how you form the belief in the first place.
\nIt made me notice my own patterns. Reading two hundred pieces of my own writing back, the recurring themes are obvious to me now and were invisible while writing: verification over generation, boring over clever, measurement over intuition, and a persistent suspicion of anything that requires you to trust rather than check.
\nI did not set out with a thesis. It assembled itself.
\nIt taught me to be wrong in public, which is a skill and is uncomfortable and is the only way to find out you were wrong quickly.
\nI graded myself in December and the pattern held for the following eight months.
\nI am reliably right about technical trajectories and reliably wrong about adoption.
\nLocal models got good; people did not switch, because hosted models got cheap faster than I expected. RAG got less necessary; the infrastructure repositioned instead of dying, which I have now failed to predict three separate times. Registry security controls were obviously needed; they arrived after the incident rather than before.
\nThe lesson I keep relearning: the technology is the easy part to forecast, and the technology was never the hard part. Human and organizational behavior is where the uncertainty lives and where the consequences land.
\nI under-predict inertia and over-predict rationality. Almost every wrong call has that shape.
\nTwo hundred pieces, roughly half of them about things that happened in a specific week.
\nReading them back, the ones that held up are almost never the ones that reported the event. They are the ones that used the event to explain a mechanism.
\nNobody needs my summary of what a company announced. They can read the announcement. What is worth writing is: why does this shape of thing keep happening, and what does it imply for what you should do on Monday.
\nThe news is a prompt. The mechanism is the article.
\nI did not know that when I started and it took about forty pieces to figure out.
\nEverything published here is still at its original URL. Nothing has been quietly edited, renamed, or removed. Corrections are marked in place.
\nThat is a deliberate choice and it costs something \u2014 there are pieces I would write differently now, and a few I think are wrong.
\nLeaving them up is the point. A publication that silently revises its history is not a record, it is a marketing surface. The value of an archive is that it shows what someone thought at the time, including when that was wrong, and you can only get that by not touching it.
\nIf you want to know whether to trust a technical writer, check whether their old pieces still exist and whether the wrong ones were corrected in the open.
\nGray background. Monospace headings. One column. No popups, no cookie banner, no newsletter modal, no autoplaying anything, no third-party JavaScript on any page.
\nThis is not minimalism as an aesthetic. It is that every one of those things was added to a website to serve the publisher at the reader's expense, and the cumulative effect has made reading on the web genuinely unpleasant.
\nA page should render before you notice it loading. Text is the interface. Nothing moves unless the reader moved it.
\nThose are not hard constraints to meet. Almost nobody meets them, and the reason is never technical.
\nSame beat. Tech, developers, and the code underneath. News where the news teaches something, essays where the news does not.
\nMore on the verification problem, because it is the defining engineering question of this period and it is nowhere near resolved. More on the craft, because the craft is what survives the tooling cycles. Fewer pieces about model launches, because they have stopped being informative.
\nThanks for reading. Corrections are always welcome and get priority over everything else.
\n\u2014 Dom
", "image": "https://readme.news/cards/two-hundred-pieces-in.png", "date_published": "2026-08-03T09:00:00Z", "date_modified": "2026-08-03T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "retrospective", "craft", "opinion" ] }, { "id": "https://readme.news/the-ai-acts-high-risk-obligations-take-effect-today/", "url": "https://readme.news/the-ai-acts-high-risk-obligations-take-effect-today/", "title": "The AI Act's high-risk obligations take effect today", "summary": "Two years after entry into force, the substantive requirements arrive. What changes, what was delayed, and what to do now.", "content_html": "The EU AI Act's obligations for high-risk AI systems apply from today. This has been scheduled since the Act entered into force in August 2024, and it is the point at which the most substantive requirements become enforceable.
\nFor providers of high-risk systems \u2014 those used in employment, education access, credit, essential services, law enforcement, migration, justice, and as safety components in regulated products:
\nDeployers \u2014 organizations using a high-risk system rather than supplying it \u2014 have a lighter but real set: use it according to instructions, ensure input data is relevant, monitor operation, retain logs, and assign competent human oversight.
\nThe implementation timeline has been adjusted more than once during the run-up, which is normal for regulation of this scope and which made planning genuinely difficult for anyone trying to comply.
\nThe important practical consequences:
\nSome obligations phase in later for systems already on the market, and for high-risk systems that are safety components of products covered by other EU legislation.
\nHarmonised standards are still being finalised. Conformity assessment is easier when you can demonstrate compliance against a standard; where standards are not yet available, providers must demonstrate compliance against the requirements directly, which is more work and more uncertain.
\nEnforcement capacity varies by member state. Market surveillance authorities are at different stages of readiness. That is not a reason to assume non-enforcement \u2014 it is a reason to expect inconsistency in the first period.
\nCheck the current state before making decisions. The details have moved and may move again.
\nIf you have not classified your systems, do that today. It is the prerequisite for everything else and it is a legal question with technical inputs. A number of organizations discover they are in scope for one feature they had not considered \u2014 a CV screening tool, a proctoring feature, a creditworthiness proxy.
\nDocumentation is the bulk of the work. Training data provenance, validation methodology, known failure modes, intended use and misuse. Most engineering teams do not have this written down and reconstructing it is slower than producing it as you go.
\nLogging is a technical requirement. Automatic recording of operation, retained, with enough detail to trace a decision. If your system does not do this, it is engineering work with a deadline that has passed.
\nHuman oversight is a design constraint, not a checkbox. \"A person reviews the output\" is insufficient if the person cannot meaningfully understand, question, or override it. Surfacing confidence, explaining the inputs that drove a decision, and making override easy and recorded are interface requirements.
\nMost software is not. The transparency obligations \u2014 disclose that users are interacting with an AI system, label synthetic media \u2014 apply much more broadly and are much lighter.
\nThe thing worth doing regardless of scope: know which category you are in, and write down why. \"We assessed this and concluded it is not high-risk because X\" is a one-page document that saves an enormous amount of time when someone asks, and someone will ask.
\nThe Act is the most comprehensive AI regulation any jurisdiction has attempted. It has real criticisms \u2014 compliance cost falls hardest on small companies, the risk categories map imperfectly onto how systems are actually built, and the standards process has lagged the deadlines.
\nIt is also law, it has extraterritorial reach, and penalties scale with global turnover.
\nThe organizations that started classification work a year ago are in reasonable shape today. The ones that waited for clarity are discovering that regulatory clarity tends to arrive after the deadline rather than before it, which is a lesson that generalizes well beyond this particular Act.
", "image": "https://readme.news/cards/the-ai-acts-high-risk-obligations-take-effect-today.png", "date_published": "2026-08-02T09:00:00Z", "date_modified": "2026-08-02T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "policy", "ai" ] }, { "id": "https://readme.news/two-years-of-agentic-coding-what-stuck/", "url": "https://readme.news/two-years-of-agentic-coding-what-stuck/", "title": "Two years of agentic coding: what stuck", "summary": "The workflows that survived contact with real work, the ones that did not, and what the whole thing actually changed.", "content_html": "Terminal coding agents went from research preview to standard tooling in about two years. Enough time has passed to separate what stuck from what was a phase.
\nMechanical refactors at scale. The clearest win, by a wide margin. Renaming a concept across four hundred files, migrating a deprecated API, converting a pattern used everywhere. Verifiable, tedious, and exactly what the tools are good at.
\nThe important second-order effect: refactors that were too expensive to do now happen. A codebase where cross-cutting cleanup is affordable is a meaningfully better codebase, and that is a permanent improvement rather than a productivity number.
\nWorking in unfamiliar territory. A language you do not know, a framework you have not used, an API you have never touched. The median output in an unfamiliar domain is better than your first attempt, and reading it teaches you the idioms.
\nThis is the use case I would defend most strongly and it is discussed least.
\nTest generation from a specification. Not \"write tests for this function\" \u2014 that produces tests that assert the implementation. But \"here is the behavior, write tests that verify it\" works well and it inverts the effort in the right direction.
\nInvestigation. Reading logs, bisecting history, tracing a call path, summarizing a large diff. Parallelizable, cheap, and it saves the expensive resource, which is your attention.
\nRepository-level instruction files. AGENTS.md and its equivalents became standard practice, and the discipline of writing down how your project actually works improved documentation for humans as a side effect.
Fully autonomous feature development. The demo works. The real version produces a plausible implementation of a subtly different feature, because the requirements that live in someone's head were never written down.
\nAgent fleets at high concurrency. The generation scales; the review does not. Two to three concurrent agents with one reviewer turned out to be the practical limit, and the constraint is entirely on the human side.
\nOrchestration frameworks. Absorbed into the models, as function-calling libraries and JSON-repair libraries were before them. The durable layer was never orchestration.
\n\"Just describe it and it builds.\" For anything with design decisions, the description that is precise enough to produce the right result is approximately as long as the code, and writing it is the same work.
\nReview is the bottleneck, permanently. Generation got roughly two orders of magnitude cheaper. Verification got no cheaper at all. Everything downstream follows from that asymmetry and nothing in two years has changed it.
\nTests became the primary artifact. If the implementation is cheap and verification is expensive, effort moves to specification. The teams getting the most out of these tools are the ones with strong test suites, and the correlation is not subtle.
\nType systems got a promotion. Every constraint the compiler checks is verification you do not perform by reading. Teams that were ambivalent about strict typing became evangelists, and the reason is always the same: it catches the class of error generated code produces most.
\nSmall diffs became non-negotiable. A machine can produce two thousand lines effortlessly. Accepting it is not a favor to anyone.
\nThe skill that separates people is judgment, not speed. It always was. It is now the only thing, and the gap between engineers who can tell when output is wrong and engineers who cannot is much more visible than it was.
\nThe apprenticeship problem.
\nThe judgment that makes a senior engineer valuable was acquired by writing a lot of code badly and then debugging it. That work is being automated. Nobody has a replacement for how the next generation acquires it, and junior hiring contracted sharply during exactly the period when the training mechanism was being removed.
\nI have written about this several times and I still do not have an answer beyond: deliberately do the hard part yourself sometimes, review generated code carefully as a learning exercise, and hire juniors anyway.
\nThat is a partial answer to a structural problem and I am not satisfied with it.
\nThese tools are genuinely useful and the useful envelope is narrower and more specific than either the enthusiasts or the skeptics claimed.
\nThey are excellent at bounded, verifiable, tedious work. They are unreliable at anything requiring judgment about what should be built. They multiply output and do not multiply throughput, because throughput is limited by review.
\nThe engineers getting the most from them are the ones who were already good at specifying problems precisely and at telling when something is wrong. That is not a new skill and it was never evenly distributed.
\nWhich is roughly what every previous tooling revolution did: raised the floor, moved the bottleneck, and made expertise more valuable rather than less.
", "image": "https://readme.news/cards/two-years-of-agentic-coding-what-stuck.png", "date_published": "2026-07-31T09:00:00Z", "date_modified": "2026-07-31T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "ai", "coding-agents", "craft", "retrospective" ] }, { "id": "https://readme.news/deleting-code-is-the-highest-value-work-nobody-schedules/", "url": "https://readme.news/deleting-code-is-the-highest-value-work-nobody-schedules/", "title": "Deleting code is the highest-value work nobody schedules", "summary": "Every line you remove is one that cannot break, cannot be misread, and does not need to be maintained.", "content_html": "The most valuable pull request I have ever reviewed removed eleven thousand lines and added forty.
\nDeleting code is the only refactoring that is unambiguously good. It cannot introduce a bug in the deleted code, because there is no deleted code. It reduces build time, test time, cognitive load, security surface, and the probability that someone reads the wrong thing.
\nAnd nobody schedules it.
\nDead code. Never called. Nobody noticed, because nothing fails when unused code exists.
\nFeatures nobody uses. Built for a customer who churned, an experiment that ended, a requirement that changed. Still there, still tested, still maintained, still appearing in every search result.
\nAbandoned abstractions. Someone built a plugin system for the second plugin, which was never written. Now every call goes through a registry that has one entry.
\nConfiguration for conditions that no longer occur. A flag for a migration that completed in 2023.
\nCompatibility shims. For a version nobody runs, an API that was removed, a browser that no longer exists.
\nTests for deleted behavior. Still running, still slow, testing something that cannot happen.
\nVendored copies. Of a library that is now a real dependency.
\nCommented-out code. Always. Delete it. Git remembers.
\nCoverage, over a long window. Run coverage in production if your language supports it, or over your full integration suite. Anything at zero across a month is a candidate.
\nBe careful: zero coverage does not prove dead. It might be an error path, a rare branch, or something exercised only in a region you did not sample. Verify before deleting.
\nStatic analysis for unreachable code. Most linters find unreferenced functions within a module. Cross-module dead code is harder and several tools do it.
\nFeature flags at 100% for over a quarter. The disabled branch is dead code with a switch on it.
\nEndpoints with no traffic. Log every route. Anything with zero requests in ninety days is a candidate. This is trivially easy to check and almost nobody does.
\nDeprecation warnings nobody triggers. If you have been logging a deprecation warning for a year and it has never fired, the deprecated thing is unused.
\ngit log on the file. Anything untouched for three years in an actively developed codebase is either perfect or forgotten.
Log before you delete. For anything you are not certain about, add logging and wait. A month of zero calls is strong evidence.
\ndef old_thing(x):\n log.warning(\"old_thing called\", stack=traceback.format_stack())\n return new_thing(x)Delete in a separate commit from any other change. A deletion mixed with a refactor is unreviewable and un-revertable.
\nDelete the tests too. Tests for deleted code are the most common thing left behind and they will confuse the next person enormously.
\nDo not comment it out. Do not move it to an old/ directory. Do not keep it \"just in case.\" Git has it. If you genuinely might need it, note the commit hash in the deletion's commit message.
Do it in batches, by area. One area, fully cleaned, is better than a thousand scattered deletions that are impossible to review.
\n\"What if we need it?\" You will not. And if you do, it is in git. In many years I have never seen a team need to recover deleted code that they could not recover.
\n\"Someone might be using it.\" Measure. That is what the logging is for. Guessing in either direction is worse than checking.
\n\"It works, why touch it?\" Because it costs. Every line is read by every person who greps this file, is compiled on every build, is scanned by every security tool, and is a possible place for a future bug.
\n\"That is not a priority.\" Correct, and it never will be, which is why it has to be scheduled rather than prioritized. Ten percent of one sprint, quarterly, with a line count as the deliverable.
\nMake it a competition. Track lines removed. Celebrate the largest deletion of the quarter.
\nThis is slightly silly and it works, because it inverts the default incentive. Engineers are implicitly rewarded for adding \u2014 features shipped, code written \u2014 and never for removing, even though removing is frequently worth more.
\nNaming it, tracking it, and praising it is enough to change the behavior.
\nA large fraction of most mature codebases is dead or effectively dead. I have never audited one where it was under 10%, and I have seen 40%.
\nEvery line of that is being read, compiled, tested, scanned, and maintained, at a cost nobody has ever measured.
\nGo find some.
", "image": "https://readme.news/cards/deleting-code-is-the-highest-value-work-nobody-schedules.png", "date_published": "2026-07-29T09:00:00Z", "date_modified": "2026-07-29T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "craft", "engineering", "opinion" ] }, { "id": "https://readme.news/the-performance-budget/", "url": "https://readme.news/the-performance-budget/", "title": "The performance budget", "summary": "A number, agreed in advance, that fails the build. The only mechanism that has ever kept a system fast.", "content_html": "Every system starts fast and gets slower. Not because of one bad decision \u2014 because of a hundred small ones, each of which added two milliseconds, none of which anyone could reasonably object to.
\nThe only mechanism I have seen reliably prevent this is a budget: a number, agreed in advance, that fails the build when exceeded.
\nBecause \"fast\" has no threshold, so there is never a moment where a specific change is the problem.
\nEvery individual addition is defensible. The library is 12 KB and saves a week. The query is 8 ms and enables a feature. The middleware is 3 ms and improves security.
\nTen of those and your page is 400 ms slower, and no single change was wrong. There was no point at which anyone could say no, because the comparison was always \"this change versus nothing\" rather than \"this change versus the budget.\"
\nA budget changes the comparison. Now the question is \"what are you willing to remove to make room for this,\" which is a real conversation.
\nDerive them from user-facing outcomes, not from what you currently have.
\nFor a web frontend:
\n| metric | budget |
|---|---|
| JavaScript, compressed | 170 KB |
| CSS, compressed | 60 KB |
| Largest Contentful Paint (p75, mobile) | 2.5 s |
| Interaction to Next Paint (p75) | 200 ms |
| Total requests, initial load | 40 |
For an API:
\n| metric | budget |
|---|---|
| p50 latency | 50 ms |
| p99 latency | 500 ms |
| database queries per request | 10 |
| memory per instance | 512 MB |
Per-layer budgets are the underrated part. A single end-to-end number tells you something regressed. Per-layer budgets tell you where.
\ntotal request budget: 200 ms\n auth: 10 ms\n validation: 5 ms\n database: 80 ms\n business: 40 ms\n serialize: 15 ms\n overhead: 50 msWhen the database layer goes to 120 ms, that specific budget fails, and the person who changed the query knows immediately rather than in a month when someone investigates a general slowdown.
\nThis is borrowed directly from game development, where per-subsystem frame budgets have been standard practice for decades.
\nThe budget must fail something, or it is a wish.
\nIn CI, on every pull request:
\n- name: bundle size\n run: npx size-limit # fails if over the configured budget\n\n- name: performance test\n run: k6 run --threshold 'http_req_duration{p(99)}<500' load.jsReport the delta on the pull request. \"This change adds 8 KB to the main bundle (142 KB \u2192 150 KB, budget 170 KB).\" Visible, in context, at the moment of decision.
\nAllow overrides with a written reason. Not blocking forever \u2014 blocking until somebody says why. The record of overrides is itself useful; if you are overriding every week, the budget is wrong or the system is losing.
\nThe single most useful backend budget, and the least common.
\nCount database queries per request in tests. Assert a maximum.
\nwith assert_max_queries(10):\n client.get(\"/api/orders\")This catches N+1 queries at the moment they are introduced, which is the only cheap time to catch them. An N+1 that reaches production is found weeks later by someone investigating a slow endpoint, and by then it is embedded in an ORM relationship that four other things depend on.
\nAlmost every framework has a way to do this, and almost nobody does.
\nThe conversation that a budget forces, in order:
\nAny of those is better than the default, which is that the change lands and the system is permanently slower.
\nBefore setting a budget, get the current numbers and the distribution. p50, p75, p95, p99. On real user hardware and real networks, not on a developer laptop on office wifi.
\nThen set the budget at roughly where you are, and ratchet it down over time rather than setting an aspirational number you fail immediately.
\nA budget you exceed on day one gets disabled on day two.
", "image": "https://readme.news/cards/the-performance-budget.png", "date_published": "2026-07-27T09:00:00Z", "date_modified": "2026-07-27T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "performance", "engineering", "craft", "frontend" ] }, { "id": "https://readme.news/what-i-look-for-in-a-codebase-in-the-first-hour/", "url": "https://readme.news/what-i-look-for-in-a-codebase-in-the-first-hour/", "title": "What I look for in a codebase in the first hour", "summary": "A checklist for assessing an unfamiliar codebase quickly \u2014 for a job, a due diligence, or a project you inherited.", "content_html": "You have an hour with an unfamiliar codebase and you need to form a judgment: is this healthy, what will it cost to work in, what is the risk.
\nHere is the order I go in, and what each thing tells you.
\nClone it. Follow the README. Start a timer.
\nThis is the single most informative test and most people skip it in favor of reading code.
\nNote every step that failed. That list is the health assessment.
\nRun it. Then look at it.
\nDoes it pass? On a clean checkout, first try. If not, that tells you the team has normalized a red build.
\nHow long? Under two minutes is excellent. Over ten and people have stopped running it locally.
\nWhat is the ratio of assertions to setup? Read three test files. If setup dominates, the code is heavily coupled.
\nAre there tests for the error paths? Almost nobody writes these and their presence is a strong positive signal.
\nIs there a skip or xfail graveyard? Count them. A pile of disabled tests means the suite has been losing a slow argument with reality.
tokei . # lines by language\ngit log --oneline | wc -l\ngit log --format='%an' | sort | uniq -c | sort -rn | headContributor concentration. If one person wrote 80% of it and they left, that is the largest risk in the codebase, larger than anything technical.
\nLanguage sprawl. Four languages in a small project usually means four sets of tooling and nobody who understands all of it.
\nDirectory structure. Does it reflect the domain or the framework? models/, views/, controllers/ tells you nothing about what the software does. billing/, inventory/, shipping/ tells you everything.
find . -name '*.py' -not -path '*/.venv/*' | xargs wc -l | sort -rn | head -20Every codebase has a few enormous files. Open the biggest one.
\ngit log -1 on it. If it is huge and changes weekly, that is the hot spot, and any work you do will touch it.How many? Compare against similar projects. An unusual number in either direction is worth understanding.
\nHow old? Anything more than two major versions behind is an upgrade project waiting for you.
\nAnything abandoned? Check the largest ones for last release date. A critical dependency with no release in three years is a fork you have not made yet.
\nAnything surprising? A cryptography library nobody has heard of. A vendored copy of something. A dependency on a specific fork.
\ngit log --oneline -50Message quality. \"fix\", \"wip\", \"asdf\" versus real descriptions. This is a direct readout of the engineering culture and it is remarkably predictive.
\nCommit size. Are they atomic, or is every commit a 3,000-line dump?
\nIs there review? Merge commits from pull requests, or direct pushes to the main branch?
\nCadence. Steady, or bursts separated by silence?
\nFrequently the most informative part.
\nCONTRIBUTING.md or equivalent on a project with multiple contributors..env.example. You cannot configure it without asking someone.Pick a user-visible feature and trace it from the entry point to the database.
\nThis is the highest-value ten minutes. You learn: how many layers, how much indirection, where the business logic lives, whether the abstractions are consistent, and whether you could add a similar feature without asking anyone.
\nIf you cannot follow it in ten minutes, neither can anyone else, and every change will be expensive.
\nAfter an hour I can usually say:
\nNone of that requires understanding what the software does. It is all structural, and structure is what determines the cost of working in something.
", "image": "https://readme.news/cards/what-i-look-for-in-a-codebase-in-the-first-hour.png", "date_published": "2026-07-24T09:00:00Z", "date_modified": "2026-07-24T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "craft", "engineering", "code-review", "careers" ] }, { "id": "https://readme.news/the-unreasonable-effectiveness-of-a-changelog/", "url": "https://readme.news/the-unreasonable-effectiveness-of-a-changelog/", "title": "The unreasonable effectiveness of a changelog", "summary": "A file that takes ten minutes per release and answers most of the questions your users would otherwise ask you.", "content_html": "Most projects do not have a changelog. Most projects have a commit log and a release page auto-generated from pull request titles, which is not the same thing and does not serve the same purpose.
\nA real changelog is a small amount of work with an outsized return.
\nDeciding whether to upgrade. The single most common reason someone reads a changelog. They are on version 3.2, version 3.7 exists, and they want to know whether it is worth the risk.
\nKnowing what will break. The most important information you can provide, and the thing auto-generated release notes are worst at.
\nDebugging. \"This started failing after we upgraded\" \u2014 a good changelog turns that into \"here is the change that caused it\" in thirty seconds.
\nFinding out what exists. People discover features by reading changelogs. This is a real and underrated distribution channel for your own work.
\nA list of merged pull request titles has three problems.
\nIt is written for the wrong audience. \"Refactor connection handling\" means something to the maintainer and nothing to the user. What changed for them?
\nIt has no hierarchy. A breaking change and a typo fix appear as sibling bullets of equal weight.
\nIt has no migration guidance. \"Remove deprecated parse() method\" tells you something broke. It does not tell you what to do about it.
Keep a Changelog is the established convention and it is good. The structure:
\n## [4.2.0] - 2026-07-22\n\n### Breaking\n- `Client.connect()` no longer accepts a positional timeout.\n Pass `timeout=` as a keyword.\n # before\n client.connect(host, 30)\n # after\n client.connect(host, timeout=30)\n\n### Added\n- `Client.ping()` for health checks without a full round trip (#412)\n- Support for Unix domain sockets via `unix://` URLs (#398)\n\n### Fixed\n- Connections leaked when the handshake timed out (#405).\n If you saw file descriptor exhaustion under load, this was it.\n\n### Deprecated\n- `Client.legacy_mode` \u2014 will be removed in 5.0. Use `compatibility=`.\n\n### Security\n- Fixed a case where credentials could appear in debug logs (GHSA-xxxx-xxxx).\n Affects 4.0.0\u20134.1.3. Rotate credentials if debug logging was enabled.Breaking changes first, always. That is what people are scanning for. Do not bury them under twelve feature bullets.
\nInclude the migration. A breaking change without \"do this instead\" makes the reader open your source code. Two lines of before-and-after saves everyone time.
\nDescribe the user-visible effect, not the implementation. Not \"refactored the retry logic.\" Rather: \"retries now use exponential backoff with jitter; if you relied on the previous fixed 1-second interval, set retry_delay=1.0.\"
Say who is affected. \"If you use X, this changes for you. Otherwise nothing changes.\" Most readers can then stop reading, which is a service.
\nLink to the issue or pull request for anyone who wants detail. The changelog is a summary, not a substitute.
\nDate every release, in ISO format. Version numbers alone do not tell you whether you are two months or three years behind.
\nWrite it as you go, not at release time. An Unreleased section at the top that each pull request adds to. Reconstructing a changelog from git history at release time is miserable and it is why changelogs get skipped.
If you fix a security issue, say so, with:
\nThat last one is the part that gets omitted and it is critical. \"Upgrade to 4.2.0\" is insufficient if credentials may have been exposed \u2014 the user also needs to rotate them, and they will not know unless you say so.
\nThe same file works for internal services, and the audience is your future self and the person who takes over the service.
\nThe most valuable internal changelog entries are the ones that record a decision:
\n## 2026-07-14\n- Switched from polling to webhooks for order status. Polling was\n costing ~40 requests/second against the vendor's rate limit and\n we were getting throttled during peaks.Six months later, when someone asks why there is webhook infrastructure, the answer is in one place.
\nTen minutes per release. In exchange:
\nThere are not many ten-minute tasks with that profile.
", "image": "https://readme.news/cards/the-unreasonable-effectiveness-of-a-changelog.png", "date_published": "2026-07-22T09:00:00Z", "date_modified": "2026-07-22T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "documentation", "open-source", "craft", "communication" ] }, { "id": "https://readme.news/property-based-testing-deserves-its-moment/", "url": "https://readme.news/property-based-testing-deserves-its-moment/", "title": "Property-based testing deserves its moment", "summary": "State an invariant, let the machine find the counterexample. A twenty-year-old technique that fits this moment exactly.", "content_html": "Example-based tests check the cases you thought of. Property-based tests check the cases you did not.
\nThe technique has been around since QuickCheck in 1999 and has stayed niche. It fits the current moment unusually well, and it is worth another look.
\nInstead of \"for this input, expect this output,\" you state a property that should hold for all inputs, and the framework generates hundreds of inputs trying to break it.
\nfrom hypothesis import given, strategies as st\n\n@given(st.lists(st.integers()))\ndef test_sort_is_idempotent(xs):\n assert sorted(sorted(xs)) == sorted(xs)\n\n@given(st.lists(st.integers()))\ndef test_sort_preserves_elements(xs):\n assert sorted(xs).count == xs.count # same multiset\n\n@given(st.text())\ndef test_roundtrip(s):\n assert decode(encode(s)) == sThe framework generates empty lists, single elements, lists with duplicates, huge values, negative numbers, and \u2014 critically \u2014 when it finds a failure, it shrinks the input to the minimal case that still fails.
\nThat shrinking is what makes the technique practical. A failure on a 400-element list is not actionable. The same failure shrunk to [0, 0] tells you exactly what is wrong.
The hard part is not the tooling. It is identifying properties, and there is a small catalogue that covers most cases.
\nRound trip. decode(encode(x)) == x. Serialization, compression, parsing, encryption. This one property catches an enormous number of real bugs and applies almost everywhere.
Invariants. Something that is always true regardless of operations. A balanced tree stays balanced. A sorted list stays sorted. Account balances sum to the same total across a transfer. A cache never returns stale data past its TTL.
\nOracle. Compare against a simpler, slower, obviously-correct implementation. You have an optimized version; write the naive one, and assert they agree. This is extremely powerful and underused \u2014 it is how you test the fast path against the version you can reason about.
\nIdempotence. f(f(x)) == f(x). Normalization, deduplication, and \u2014 importantly \u2014 any API operation that claims to be idempotent. If you have idempotency keys, this is how you actually verify them.
Commutativity and associativity. Order should not matter. Merge operations, set operations, CRDT merges.
\nMetamorphic relations. When you cannot state the correct output, state how the output should change. Adding an item to a cart should increase the total by that item's price. Searching for a more specific query should return a subset. Sorting descending should be the reverse of sorting ascending.
\nThis last category is the one that unlocks property testing for business logic, where there is no obvious oracle.
\nParsers and serializers. Round trip, always.
\nData structures. Invariants after every operation sequence.
\nState machines. Generate random valid operation sequences, assert the invariants hold throughout. This finds ordering bugs that no hand-written test would.
\nAnything with an obvious naive implementation. Oracle testing.
\nFinancial and unit arithmetic. Rounding, currency, conversions. The edge cases are numerous and boring, which is exactly what a generator is for.
\nConcurrent code, with a framework that generates interleavings. This is the hardest category to test any other way.
\nUI. The properties are aesthetic.
\nAnything where the correct output requires human judgment.
\nIntegration tests against real systems. Generation implies many runs, and many runs against a real database is slow.
\nCode where you cannot state a property. If you genuinely cannot, that may itself be information about the design \u2014 code with no statable invariants is code with no contract.
\nDo not replace example tests. Keep them. They document intent and they are the fastest way to communicate what a function is for. Add properties alongside.
\nStart with round-trip properties. Easiest to state, highest hit rate for real bugs.
\nSave the failing seed. When a property fails, the framework gives you the counterexample. Add it as a regression example test so it is checked deterministically forever.
\nBound the input space. Unbounded generation produces absurd inputs and slow tests. Constrain to realistic ranges: strings up to a reasonable length, numbers in a plausible domain.
\nRun more cases in CI than locally. A hundred examples locally for fast feedback, a thousand in CI, ten thousand in a nightly run.
\nTwo reasons this fits the current moment.
\nVerification is the bottleneck. Property testing is verification you write once and that checks a large input space forever. That is exactly the leverage the moment calls for.
\nStating a property is a task worth doing carefully by hand, and generating the implementation is not. Writing \"the total must always equal the sum of line items\" requires understanding the domain; nothing else in the test does.
\nThat is a good division of labor, and it points at where human effort should concentrate: on specifying what must be true, and letting everything else be derived.
", "image": "https://readme.news/cards/property-based-testing-deserves-its-moment.png", "date_published": "2026-07-20T09:00:00Z", "date_modified": "2026-07-20T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "testing", "craft", "engineering", "type-systems" ] }, { "id": "https://readme.news/vendor-lock-in-an-honest-cost-model/", "url": "https://readme.news/vendor-lock-in-an-honest-cost-model/", "title": "Vendor lock-in: an honest cost model", "summary": "Portability is not free and neither is dependence. A framework for deciding how much abstraction to buy.", "content_html": "\"Avoid vendor lock-in\" is treated as self-evidently good advice. It is not advice, it is a preference, and following it uncritically produces systems that are worse in exchange for optionality nobody will ever exercise.
\nHere is a way to actually decide.
\nSwitching cost. How much engineering time to move off, if you had to.
\nPricing power. A vendor who knows you cannot leave prices accordingly. This is real and it is usually the largest ongoing cost.
\nCapability ceiling. You are limited to what they support, on their timeline.
\nCorrelated risk. They have an outage, you have an outage. They change their terms, you comply. They get acquired and sunset the product, you migrate on their schedule.
\nThis is the half that gets ignored, and it is frequently larger.
\nThe abstraction layer itself. Code to write, maintain, test, and debug through. It is a permanent tax and it makes stack traces longer.
\nLowest common denominator. Your abstraction can only expose what all candidate providers support. You give up the features that made the good option good.
\nThe abstraction is usually wrong anyway. It was designed against one provider's model. When you actually try to swap, you discover the abstraction encoded assumptions that do not hold, and you rewrite it.
\nDelayed value. Time spent on portability is time not spent on the product.
\nFor each dependency, estimate:
\nThen: if switching_cost \u00d7 probability < abstraction_cost, do not abstract.
The numbers are rough. The exercise still clarifies, because it forces you to state the probability out loud, and stated probabilities are usually much lower than the implied ones people are acting on.
\nObject storage. Switching cost: low. The S3 API is a de facto standard and every provider implements it. Probability: moderate \u2014 people do move for pricing.
\nVerdict: use the S3 API, do not abstract further. The API is already the abstraction.
\nCompute. Switching cost: moderate if containerized, high if you use provider-specific serverless. Probability: low.
\nVerdict: containerize \u2014 which is good practice anyway \u2014 and use whatever managed service you want. Do not build a compute abstraction layer.
\nRelational database. Switching cost: high. Probability: low.
\nVerdict: use the database's features. Teams that avoid stored procedures, database-specific types, and advanced indexing to stay portable are giving up real capability for an event that will not happen. Postgres-specific SQL is fine. You are not going to migrate to a different engine, and if you do, the SQL dialect will be the smallest part of the pain.
\nManaged queues, streams, and similar. Switching cost: moderate. The semantics differ enough between providers that a thin abstraction genuinely helps.
\nVerdict: a thin interface \u2014 publish, subscribe, ack \u2014 is worth it. Not a full abstraction; a boundary.
\nAuthentication. Switching cost: very high \u2014 you have to migrate user credentials, sessions, and integrations. Probability: low, but the consequences of being stuck are severe.
\nVerdict: use standard protocols. OIDC and SAML are the abstraction. A provider that supports them is replaceable in principle; one with a proprietary SDK is not.
\nAI model providers. Switching cost: low if you kept the interface thin. Probability: high \u2014 the market is moving fast and the right choice changes quarterly.
\nVerdict: definitely abstract. This is the clearest case on the list. A thin interface plus an eval harness in your own repository makes model changes an afternoon. Teams that did this in 2024 have been switching providers casually ever since.
\nObservability. Switching cost: moderate to high \u2014 instrumentation is everywhere in your code. Probability: moderate, usually driven by cost.
\nVerdict: OpenTelemetry. The instrumentation is vendor-neutral, the collector handles routing, and swapping backends is a configuration change.
\nAbstract where the switching probability is high and the abstraction is cheap. AI providers, observability backends, object storage.
\nDo not abstract where switching is unlikely and the abstraction costs you real capability. Databases, compute platforms, managed services you chose for their specific features.
\nUse standard protocols wherever they exist. OIDC, S3, OpenTelemetry, SQL. A standard is an abstraction someone else maintains, and it is always cheaper than yours.
\nNot an abstraction layer. Your data, in a format you can export, and a documented process for leaving.
\nAsk, before adopting anything: can I get all my data out, in a usable format, without their cooperation? If yes, you have real optionality regardless of how coupled your code is, because the expensive part of a migration is never the code \u2014 it is the data.
\nIf the answer is no, that is a much bigger red flag than any API coupling, and it is the question almost nobody asks during procurement.
", "image": "https://readme.news/cards/vendor-lock-in-an-honest-cost-model.png", "date_published": "2026-07-17T09:00:00Z", "date_modified": "2026-07-17T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "architecture", "devops", "engineering", "opinion" ] }, { "id": "https://readme.news/connection-pooling-explained-properly/", "url": "https://readme.news/connection-pooling-explained-properly/", "title": "Connection pooling, explained properly", "summary": "Why your database has 400 connections, why that is bad, and how to size a pool without guessing.", "content_html": "Connection pool sizing is done by copying a number from a blog post, and the number is usually wrong in a specific and expensive way.
\nIn Postgres specifically, each connection is a separate operating system process with its own memory. A few megabytes of baseline, plus work memory for sorting and hashing, plus its share of shared buffer access.
\nFour hundred connections means four hundred processes. The scheduler is context switching between them, they are contending for the same locks and buffers, and the memory is largely wasted because most of them are idle.
\nThe counterintuitive result, which has been measured many times: throughput frequently goes down as connection count goes up, past a fairly low threshold.
\nMore connections does not mean more concurrency. It means more contention.
\nA widely used starting formula:
\nconnections = (core_count \u00d7 2) + effective_spindle_countFor an 8-core machine with SSD storage, that is somewhere around 16 to 20.
\nThat number seems shockingly low to people running pools of 100 or more. It is correct, and the reasoning is straightforward: a query is either using CPU or waiting on I/O. You need enough connections to keep the cores busy and to have some work queued behind I/O waits. Past that, additional connections are queued at the database instead of queued in your pool, and queueing at the database is worse because it consumes resources.
\nTest it. Take your load test, run it at pool sizes of 10, 20, 40, 80, and 160, and plot throughput and p99 latency. The curve rises, flattens, and then degrades. Most people are on the degrading side and have never looked.
\nThree, and they do different things.
\nApplication-side pool. In-process, reuses connections across requests. Every ORM and database driver has one. This is the minimum.
\nExternal pooler \u2014 PgBouncer, pgcat, or a cloud provider's equivalent. Sits between your application and the database and multiplexes many client connections onto few server connections.
\nThis is what you need when you have many application instances. Twenty instances with a pool of 20 each is 400 connections to the database, even if each instance is mostly idle. A pooler collapses that to the number the database actually wants.
\nThe database's own limit. max_connections. Set it lower than you think \u2014 it is a safety valve, and setting it high does not make the database faster, it makes the failure mode worse.
External poolers have modes and choosing wrong causes subtle correctness bugs.
\nSession pooling. A client gets a server connection for the duration of its session. Safe, and provides little multiplexing benefit.
\nTransaction pooling. A server connection is assigned per transaction and returned after commit. This is where the big multiplexing win is, and it is what most people want.
\nThe catch: anything that depends on session state breaks.
\nSET at the session levelLISTEN/NOTIFYWITH HOLD cursorsIf your ORM uses server-side prepared statements by default \u2014 many do \u2014 you must either disable them or use a pooler that handles them. This is the single most common transaction-pooling problem and it manifests as intermittent errors under load, which is a miserable thing to debug.
\nStatement pooling. A connection per statement. Maximum multiplexing, breaks multi-statement transactions. Almost never what you want.
\nBeyond size:
\nConnection timeout. How long a request waits for a connection before failing. This should be short \u2014 a few seconds. A request waiting thirty seconds for a connection has already been abandoned by its caller.
\nIdle timeout. Return connections to the database when not in use. Important with many application instances.
\nMax lifetime. Recycle connections periodically. This handles the case where a database failover happens and your pool is holding connections to the old primary \u2014 without a max lifetime, some pools hold those indefinitely.
\nValidation. Test a connection before handing it out, or handle the failure on first use. Something has to deal with connections that died while idle.
\nSELECT state, count(*), max(now() - state_change) AS longest\nFROM pg_stat_activity\nWHERE backend_type = 'client backend'\nGROUP BY state;What you are looking for:
\nidle in transaction \u2014 the worst state. A transaction is open and doing nothing, holding locks and preventing vacuum. This is an application bug: a transaction opened and not committed, usually because of an early return or an exception path.idle \u2014 pool is oversized. Harmless but wasteful.active with long durations \u2014 queries are slow; the pool is not the problem.idle in transaction is the one to alert on. It is always a bug and it causes outages that look like database problems and are application problems.
Size your pool from a load test, not from a blog post. It is smaller than you think. Use an external pooler in transaction mode if you have many application instances, and check your prepared statement behavior when you do.
", "image": "https://readme.news/cards/connection-pooling-explained-properly.png", "date_published": "2026-07-15T09:00:00Z", "date_modified": "2026-07-15T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "databases", "performance", "architecture" ] }, { "id": "https://readme.news/the-onboarding-document-that-actually-works/", "url": "https://readme.news/the-onboarding-document-that-actually-works/", "title": "The onboarding document that actually works", "summary": "Most onboarding docs are written by people who already know. Here is what a new engineer needs on day one, in order.", "content_html": "Onboarding documentation is written by someone who already knows the system, which means it is written from the wrong side of the knowledge gap.
\nThe result is consistently: an architecture overview that is meaningless without context, a list of tools with no explanation of why, and no answer to any question a new person actually has.
\nDay one: get something running.
\nNot understanding. Running. A new engineer who has the application running locally on day one is in a completely different position from one who is still fighting a dependency on day three.
\nThis section should be a numbered list of commands that work. Tested. On a clean machine. Recently.
\n## get it running\n\n1. Install prerequisites: `brew bundle` (or the equivalent \u2014 see below)\n2. Copy the environment: `cp .env.example .env`\n3. Get secrets: `./scripts/fetch-dev-secrets` (needs VPN)\n4. Start dependencies: `docker compose up -d`\n5. Migrate: `./scripts/db-reset`\n6. Run: `pnpm dev`\n\nYou should see the app at http://localhost:5173 with seeded data.\nLog in with dev@example.com / password.\n\nIf step 3 fails with \"unauthorized\", you need to be added to the\ndev-secrets group \u2014 ask in #eng-onboarding.That last paragraph \u2014 the anticipated failure with its resolution \u2014 is the part that separates a document that works from one that does not. Every step that has ever failed for anyone should have a note.
\nDay two: make a change and see it.
\nA guided first change. Something trivial and real: change a label, add a field, fix a typo in a template. All the way through: edit, test, review, merge, deploy.
\nThe point is not the change. It is that they have now executed the entire delivery pipeline once, so every subsequent change is a variation on something they have done.
\nDay three to five: the map.
\nNow the architecture overview, and now it means something, because they have seen the system run.
\nKeep it to a page. What are the major pieces, what does each do, how do they talk. A diagram. Where the code for each piece lives.
\nNot a complete description. A map, at the resolution of \"which building do I go to.\"
\nWeek two: the why.
\nThe decisions that are not obvious from the code. Why the database is structured that way. Why there is a queue between those two services. Why that module is frozen. Why the obvious approach to X does not work.
\nThis is the highest-value and least-written documentation in any organization, because it exists only in the heads of people who were there. When they leave, it is gone, and the next person spends a year rediscovering it \u2014 usually by proposing the obvious approach and being told no.
\nThe glossary. Every organization has jargon: internal product names, acronyms, words used with a specific local meaning. A new person hears twenty of these in their first week and cannot ask about all of them without feeling stupid.
\nWrite them down. This is a thirty-minute task with an outsized payoff.
\nWho to ask about what. Not the org chart. \"Payments: ask Priya. Deploy pipeline: #platform-help. Anything about the legacy importer: Marcus, and be warned it is complicated.\"
\nThe things that will surprise you. The test suite that fails on the first run until you seed a fixture. The service that takes four minutes to start. The one flaky test everyone knows about. The staging environment that resets on Sundays.
\nEvery codebase has these. Writing them down converts \"this is broken and I do not want to admit I cannot fix it\" into \"oh, that is expected.\"
\nWhat not to touch. Frozen modules, generated files, anything requiring a specific review.
\nOnboarding docs rot faster than any other documentation, because the people who would notice the errors are the ones who no longer read it.
\nThe fix: the last person onboarded owns it.
\nEvery new engineer's first task is to follow the document, fix everything that was wrong, and add the things they had to ask about. Then they own it until the next person arrives.
\nThis works because their frustration is fresh and their perspective is exactly the target reader's. It is the only mechanism I have seen that keeps these documents accurate.
\nHand it to a new engineer and do not help them.
\nTime how long until they have the application running. Note every question they had to ask. Each question is a gap, with a measured cost.
\nThen fix them, and repeat with the next person.
\nMost teams have never done this and would be surprised by the result.
", "image": "https://readme.news/cards/the-onboarding-document-that-actually-works.png", "date_published": "2026-07-13T09:00:00Z", "date_modified": "2026-07-13T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "documentation", "engineering-management", "craft", "careers" ] }, { "id": "https://readme.news/why-your-container-image-is-14-gigabytes/", "url": "https://readme.news/why-your-container-image-is-14-gigabytes/", "title": "Why your container image is 1.4 gigabytes", "summary": "It should be forty megabytes. Here is where the rest of it came from and how to get it back.", "content_html": "A container image for a compiled service should be tens of megabytes. For an interpreted one, low hundreds. If yours is over a gigabyte, something specific went wrong and it is usually one of six things.
\nSize matters for real reasons: pull time on cold start, registry cost, deployment speed when you are scaling out under load, and attack surface \u2014 every package in the image is something that can have a CVE you have to answer for.
\n1. You shipped the build toolchain.
\nThe compiler, the headers, the package manager cache, the source tree, the test fixtures. All of it needed to build, none of it needed to run.
\nMulti-stage builds fix this completely:
\nFROM golang:1.24 AS build\nWORKDIR /src\nCOPY go.mod go.sum ./\nRUN go mod download\nCOPY . .\nRUN CGO_ENABLED=0 go build -ldflags=\"-s -w\" -o /app ./cmd/server\n\nFROM gcr.io/distroless/static-debian12\nCOPY --from=build /app /app\nENTRYPOINT [\"/app\"]Final image: the binary, plus CA certificates and timezone data. Tens of megabytes.
\n2. You started from a full distribution image.
\nFROM ubuntu is roughly 80 MB before you install anything, and it includes a package manager, a shell, and a hundred utilities you will never invoke.
The ladder, from largest to smallest:
\n-slim variants \u2014 30\u201380 MBscratch \u2014 nothing at all, for static binariesThe Alpine caveat, since it bites people: musl's allocator and DNS resolver behave differently from glibc's. Python performance in particular can be significantly worse, and some binary wheels do not exist for musl. Test rather than assume.
\n3. Your layers are ordered wrong.
\nEach instruction creates a layer. A layer is invalidated when it or anything before it changes.
\n# bad \u2014 any source change reinstalls every dependency\nCOPY . .\nRUN npm ci\n\n# good \u2014 dependencies are cached until the lockfile changes\nCOPY package.json package-lock.json ./\nRUN npm ci\nCOPY . .This does not shrink the final image but it dramatically speeds up builds, which is usually what people actually care about.
\n4. You deleted things in a later layer.
\nRUN apt-get install -y build-essential # layer 1: +400 MB\nRUN apt-get remove -y build-essential # layer 2: marks deleted, image unchangedLayers are additive. Deleting a file in a later layer hides it and does not remove it. The bytes are still in the image and still transferred on pull.
\nEverything must happen in one RUN:
RUN apt-get update \\\n && apt-get install -y --no-install-recommends build-essential \\\n && make \\\n && apt-get purge -y build-essential \\\n && apt-get autoremove -y \\\n && rm -rf /var/lib/apt/lists/*Better: use a multi-stage build and do not install the toolchain in the final image at all.
\n5. You have no .dockerignore.
COPY . . copies .git, node_modules, build artifacts, test fixtures, and your local .env.
.git\nnode_modules\ndist\n*.log\n.env*\n**/__pycache__\ncoverageThe .git directory alone is frequently hundreds of megabytes on a mature repository, and it is in a lot of images.
6. Your dependencies are enormous.
\nSometimes it is genuinely the dependencies \u2014 machine learning stacks with CUDA libraries are legitimately multiple gigabytes.
\nCheck whether you need the GPU variant. torch with CUDA is roughly 2.5 GB; the CPU build is a fraction of that. If you are serving on CPU, you are shipping GPU libraries for nothing.
docker history --no-trunc <image> # size per layerOr use a layer inspection tool that shows you which files are in which layer and how much space is wasted. Ten minutes with one of those tells you exactly what to fix.
\nEvery package in the image is potential CVE surface, and your scanner will report all of them regardless of whether the code is reachable.
\nA distroless image has almost nothing to report, which means the reports you do get are signal rather than noise. That is worth more than the size reduction \u2014 a vulnerability report with three entries gets read; one with four hundred does not.
\nThe trade-off: no shell means you cannot docker exec in to debug. Use ephemeral debug containers that attach to the running pod's namespaces instead, which is a better practice anyway because it means your production image is not a debugging toolkit.
Most secret management advice is a product comparison. Here is the practice instead, ordered by what to do first.
\nThe single highest-value change, and it is architectural rather than a tool.
\nA static credential can be stolen, leaked, committed, logged, or exfiltrated from a developer laptop. A credential that lives for fifteen minutes and is scoped to one operation cannot be usefully stolen.
\nWorkload identity. Your CI job, your container, your function proves its identity to the cloud provider and receives a short-lived token. No stored secret at all.
\n# GitHub Actions with OIDC \u2014 no stored cloud credentials\npermissions:\n id-token: write\nsteps:\n - uses: aws-actions/configure-aws-credentials@v4\n with:\n role-to-assume: arn:aws:iam::123456789012:role/deploy\n aws-region: us-east-1That eliminates the most commonly stolen class of credential entirely. If you do one thing from this article, do this one.
\nDatabase credentials can work the same way \u2014 IAM authentication, or dynamically generated credentials from a secret manager with a short lease.
\nNot in version control. Obvious, universally violated. Run a scanner in pre-commit and in CI. Both \u2014 pre-commit catches it before it happens, CI catches it when someone skipped the hook.
\nOnce committed, assume compromised. Rewriting history does not help; the object is in every clone and in every fork. Rotate, then clean up.
\nNot in the container image. Layers are inspectable. docker history and a layer extraction tool will find it.
Not in environment variables, ideally. This is more controversial. Environment variables leak: into crash dumps, into child processes, into /proc, into logs when someone prints the environment for debugging, into error tracking services that capture context.
Files with restrictive permissions, mounted at runtime, are better. Environment variables are convenient and are the pragmatic choice for many systems \u2014 just know what you are accepting.
\nNot in logs. Redact at the logger, with a deny-list of key names, not at each call site. Somebody will forget at a call site. The logger never forgets.
\nREDACT = {\"password\", \"token\", \"secret\", \"api_key\", \"authorization\", \"cookie\"}\n\ndef scrub(d):\n return {k: (\"***\" if k.lower() in REDACT else v) for k, v in d.items()}Not in error tracking. Most error trackers capture local variables and request headers by default. Configure the redaction, then verify it by triggering a test error and reading what arrived.
\nRotation is only real if it has been executed. A rotation procedure that has never run is a document, not a control.
\nThe property that makes rotation painless: support two valid credentials at once. Add the new one, deploy, verify, remove the old one. Without overlap, rotation requires downtime, so it does not happen.
\nDesign for this when you create the credential, not when you need to rotate it under incident pressure.
\nA credential should permit exactly what its holder needs.
\nThe test: if this credential leaked, what could an attacker do? If the answer is \"anything,\" the scope is wrong regardless of how well you protect it.
\nAn inventory of every secret: what it is, where it lives, what it grants, who owns it, when it was last rotated.
\nMost organizations cannot produce this, which means they cannot answer \"what do we rotate\" during an incident, which turns a two-hour response into a two-day one.
\nWhen a secret is exposed, in this order:
\nThe common mistake is investigating first. While you investigate, the credential is live.
\nEvery major cloud has a secret manager. They are all adequate. The dedicated tools add dynamic credential generation, fine-grained policy, and audit \u2014 genuinely useful at scale and not where to start.
\nStart with: OIDC workload identity for machine-to-machine, a cloud secret manager for what remains, scanning in CI, redaction in the logger, and a tested rotation procedure.
\nThat covers the overwhelming majority of real-world credential compromise, and it is a week of work rather than a platform migration.
", "image": "https://readme.news/cards/secrets-management-practically.png", "date_published": "2026-07-08T09:00:00Z", "date_modified": "2026-07-08T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "security", "infrastructure", "engineering", "craft" ] }, { "id": "https://readme.news/code-review-comments-that-change-things/", "url": "https://readme.news/code-review-comments-that-change-things/", "title": "Code review comments that change things", "summary": "Most review comments are noise or nitpicks. A small taxonomy of the ones that are worth writing.", "content_html": "Most code review comments do not change the code, and of the ones that do, most change something that did not matter.
\nHere is a taxonomy of comments worth writing, roughly in descending order of value.
\n\"This will break when X.\" The highest-value comment there is. A specific failure scenario the author did not consider.
\n\n\"If two requests hit this concurrently, both will pass the existence check and both will insert. We saw this exact bug in the invoicing path last year.\"
Concrete, falsifiable, and it comes with evidence.
\n\"This contradicts how we do it elsewhere.\" Consistency has real value and the author frequently does not know the precedent exists.
\n\n\"
orders/uses the repository pattern for this. Worth matching, or is there a reason to differ here?\"
Note the question at the end. Sometimes there is a reason and you have just learned something.
\n\"I don't understand this.\" Underrated. If a reviewer with context cannot follow it, a stranger in two years will not either.
\nThis is not an admission of inadequacy. It is a measurement of the code's clarity, and it is a measurement only a reader can take.
\n\"What happens if this fails?\" The most consistently productive question in code review. Error paths are the least-considered part of most changes and the most-exercised part in production.
\n\"Is this the right layer?\" Business logic in a controller, presentation logic in a model, a database call in a template. Structural, cheap to fix now, expensive later.
\n\"This is good and here is why.\" Genuinely valuable and almost never written. Naming what worked teaches the pattern and it makes the critical comments land better, because they arrive from someone who is paying attention rather than someone who is looking for problems.
\nAnything a formatter or linter handles. If you are commenting on spacing, quotes, or import order, fix your tooling instead. A human enforcing mechanical rules is a broken process.
\nStyle preferences without a reason. \"I would have used a map here\" is not a review comment, it is a preference. If there is a reason \u2014 clarity, performance, consistency with a convention \u2014 say the reason. If there is not, do not send it.
\nSpeculative generality. \"What if we later need to support multiple currencies?\" Usually they will not, and building for it costs now. If you genuinely believe it, say what makes you believe it.
\nA redesign in a review comment. If the approach is fundamentally wrong, that is a conversation, not a comment thread. Comments are for improving an approach; a different approach needs a discussion, and doing it in review comments after the work is done is the most expensive possible time.
\nThat failure is on the process, not the reviewer: the design should have been discussed before implementation.
\nDistinguish blocking from non-blocking. A convention that removes an enormous amount of ambiguity:
\nblocking: this deletes rows without the tenant filter\nsuggestion: this could use the existing helper in utils/dates\nquestion: is the retry here intentional given the caller already retries?\nnit: typo in the comment\npraise: nice \u2014 this handles the empty case correctly, which the old one did notThe author knows exactly what to act on. The reviewer can leave a thought without implying it must be addressed. Both people save time.
\nAsk rather than assert when you are unsure. \"Why does this need a lock?\" is better than \"this does not need a lock\" when you are not certain, and it is better even when you are, because the answer might teach you something about the system.
\nExplain the why, not just the what. \"Use a set here\" is an instruction. \"Use a set here \u2014 this is O(n\u00b2) on a list that can have thousands of entries\" is teaching, and the author will apply it next time without being told.
\nComment on the code, not the person. \"This is confusing\" rather than \"you wrote this confusingly.\" Small difference, and it consistently changes how the comment is received.
\nThe diff is too large. Past a few hundred lines, review quality collapses. The comment \"this PR is too large, can you split it\" is the most valuable one available and it is socially costly to write, which is why nobody does.
\nMake it a policy so it is not a personal judgment.
\nReview is too late. If a fundamental problem is found in review, the process failed earlier. That belongs in design.
\nOnly one person reviews. Different reviewers see different things. For anything significant, two, with different backgrounds.
\nComments arrive over three days. The author has moved on and has to reload the entire context. Batch your review into one pass, and do it within a day.
\nThe purpose of code review is not to find bugs \u2014 tests find bugs more reliably and more cheaply.
\nIt is to spread understanding, maintain coherence, and catch the class of problem that automation cannot see: wrong abstraction, wrong layer, missing case, code that will confuse the next reader.
\nComments that serve those are worth writing. Everything else is noise with a notification attached.
", "image": "https://readme.news/cards/code-review-comments-that-change-things.png", "date_published": "2026-07-06T09:00:00Z", "date_modified": "2026-07-06T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "code-review", "craft", "communication", "engineering" ] }, { "id": "https://readme.news/backpressure-is-the-concept-your-system-is-missing/", "url": "https://readme.news/backpressure-is-the-concept-your-system-is-missing/", "title": "Backpressure is the concept your system is missing", "summary": "When a fast producer meets a slow consumer, something has to give. Deciding what, in advance, is the whole discipline.", "content_html": "Every system with a producer and a consumer eventually has a moment where the producer is faster. What happens next is either a design decision you made or an emergent behavior you discover during an incident.
\nThere are only four. Every system picks one, explicitly or by accident.
\n1. Buffer. Queue the excess.
\nWorks for bursts. Fails for sustained overload, because a buffer is a delay, and an unbounded buffer is a memory leak with a friendly name. The failure mode is that memory grows until the process dies, taking the buffer with it \u2014 so you lose everything, at the worst possible moment.
\n2. Drop. Discard the excess.
\nCorrect more often than people are comfortable with. Metrics, logs, telemetry, non-critical events \u2014 dropping 5% of samples under load is fine and dying is not.
\nThe requirement: know that you dropped, and how much. Silent drops are how you get a dashboard that looks healthy while data is missing.
\n3. Block. Make the producer wait.
\nThis is real backpressure. The consumer's slowness propagates upstream, the producer slows down, and the system reaches equilibrium at the consumer's rate.
\nCorrect for internal pipelines where the producer can wait. Dangerous when the producer is a user-facing request handler, because now user requests are blocked on a background process.
\n4. Reject. Tell the producer no.
\nThe right answer at a service boundary. A 429 or 503 returned in one millisecond is much better than a request that waits thirty seconds and then times out, because the caller can make a decision \u2014 retry later, degrade, or tell the user.
\nMost systems buffer by default, unboundedly, without anyone deciding.
\nThe failure mode is always the same: latency grows, memory grows, and then the process dies. And the requests you were holding were abandoned by their callers ten seconds earlier, so all that work was for nothing.
\nA queue that is always full is not a buffer. It is a delay you cannot see.
\nEvery queue has a maximum size. Every one. Choose the number by asking: how long should a request be willing to wait? Multiply by the consumer's rate. That is your queue depth.
\nIf you cannot answer that question, the queue is not designed.
\nPrefer rejecting to queueing at the edge. When a request arrives and the system is saturated, reject fast. The caller has a timeout; use it as your budget.
\nPropagate deadlines. If the caller has 200 ms left, every downstream operation should know that. Work performed after the caller has given up is pure waste, and under overload it is the majority of the work being done.
\nctx, cancel := context.WithTimeout(ctx, remaining)\ndefer cancel()Shed by priority, not randomly. Under load, serve health checks, serve authenticated users, serve the critical path. Shed background work, analytics, and prefetches. Random shedding means your health checks fail and your orchestrator kills healthy instances, which is a self-inflicted outage.
\n**Measure queue age, not depth.** Depth tells you how many. Age tells you how far behind. \"The oldest item has been waiting four minutes\" is actionable in a way that \"there are 30,000 items\" is not.
\nLittle's Law: L = \u03bbW. Items in the system equals arrival rate times time in system.
Rearranged: wait time equals queue length divided by service rate.
\nIf your queue holds 10,000 items and you process 100 per second, the newest item waits 100 seconds. That is not a hypothetical \u2014 it is arithmetic, and it means your queue length choice is your latency choice, whether or not you framed it that way.
\nPick the latency you can accept, multiply by the service rate, and that is your maximum queue length. Everything past it gets rejected.
\nPoint a load generator at your service at three times its capacity. Watch:
\nThat last one is the important one. A system that takes twenty minutes to recover from a two-minute overload has a backpressure problem, and it will turn a small incident into a large one on a day you did not choose.
", "image": "https://readme.news/cards/backpressure-is-the-concept-your-system-is-missing.png", "date_published": "2026-07-03T09:00:00Z", "date_modified": "2026-07-03T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "architecture", "reliability", "engineering" ] }, { "id": "https://readme.news/the-crawler-tolls-one-year-on/", "url": "https://readme.news/the-crawler-tolls-one-year-on/", "title": "The crawler tolls, one year on", "summary": "Default blocking and pay-per-crawl changed who can read the web. An assessment of what actually happened.", "content_html": "A year ago today a CDN sitting in front of a large fraction of the web flipped its default: AI crawlers blocked unless explicitly allowed, with a marketplace for charging per crawl.
\nEnough time has passed to say what actually happened rather than what everyone predicted.
\nThe norm inverted. Before, crawling was permitted by default and robots.txt was a request. Now, for a large share of the web, crawling is denied by default and access is a negotiation.
That is a genuine structural change to how the web works and it happened through one company's configuration default rather than through any standards process, legislation, or public deliberation.
\nLicensing deals concentrated. Large AI companies negotiated bulk access with large publishers. That was always the likely outcome: the parties with lawyers and leverage made arrangements, and the arrangements are private.
\nSmall publishers got very little. The pay-per-crawl mechanism works, technically. The revenue for a site with modest traffic is negligible \u2014 the arithmetic never supported anything else. The publishers who most needed a new economic model got the one that pays least.
\nNon-commercial crawling got harder. Academic researchers, archivists, and independent search projects have no licensing budget and no negotiating position. Carve-outs exist and are discretionary, which means the ability to study the web is now something you apply for.
\nThat is the outcome I was most worried about and it is the one that materialized most clearly.
\nTraining data supply. The frontier labs have enormous existing corpora, licensed sources, and synthetic generation. The marginal value of newly crawled web text was already declining. Restricting it did not create the leverage publishers hoped for.
\nTraffic. Referral traffic from search to publishers continued its decline, driven by AI answers in search results, which is a completely separate mechanism from training crawlers and which blocking crawlers does nothing about.
\nThis is the part that was most misunderstood at the time. The traffic problem and the training problem have different causes and blocking crawlers only addresses one of them \u2014 the one with less economic impact.
\nInfrastructure defaults are policy. A configuration change at a chokepoint reshaped access to a large fraction of the web, with no process and no appeal.
\nThat is not a criticism of the specific decision, which was popular and defensible. It is an observation about where power actually sits, and it generalizes: the entities that can change the web's behavior are the ones with concentration at a layer everyone depends on, and there are about five of them.
\nFor your own site, the decision remains yours and it is worth making deliberately rather than accepting a default:
\nCheck what you are actually blocking. A meaningful number of sites blocked their own search indexing in the first months of this and did not notice for weeks.
\nThe web's economic model \u2014 publish freely, get traffic, monetize traffic \u2014 is breaking, and nothing has replaced it.
\nCrawler tolls are not the replacement; the arithmetic does not work at the scale of the actual web, where most content is made by people with no ability to negotiate anything.
\nLicensing deals are not the replacement either; they work for a few hundred large publishers and for nobody else.
\nI do not know what the replacement is. I am increasingly convinced that nobody does, and that the interval between the old model failing and a new one existing is going to be long and is going to be bad for the open web.
\nThat is a genuinely pessimistic conclusion and I have not found a way around it in a year of thinking about it.
", "image": "https://readme.news/cards/the-crawler-tolls-one-year-on.png", "date_published": "2026-07-01T09:00:00Z", "date_modified": "2026-07-01T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "frontend", "ai", "policy", "infrastructure", "review" ] }, { "id": "https://readme.news/cross-platform-is-a-promise-you-make-to-your-budget/", "url": "https://readme.news/cross-platform-is-a-promise-you-make-to-your-budget/", "title": "Cross-platform is a promise you make to your budget", "summary": "Write once, run anywhere, debug everywhere. The honest accounting of what each approach actually costs.", "content_html": "The cross-platform question \u2014 one codebase or several \u2014 gets argued as a technical matter and is mostly an organizational one. Here is the honest accounting.
\nNative gives you: full platform capability, best performance, platform-idiomatic interface, immediate access to new OS features, and the best debugging tools.
\nIt costs you: two or three codebases, two or three teams, features implemented multiple times, and behavior that diverges over years in ways nobody tracks.
\nCross-platform gives you: one codebase, one team, features implemented once, consistent behavior.
\nIt costs you: a framework layer between you and the platform, a lag before new OS features are available, worse debugging when the problem is in the bridge, and an interface that is either non-idiomatic on every platform or requires platform-specific work anyway.
\nThe largest cost of cross-platform is not performance. It is the escape hatch.
\nEverything is fine until you need something the framework does not support. Then you are writing platform-specific native code, plus a bridge, plus a fallback, plus tests for all three \u2014 and you have the complexity of native development plus the framework.
\nThis happens. It always happens. The question is how often, and that depends entirely on what your app does.
\nLow escape-hatch pressure: content, commerce, forms, dashboards, CRUD, most business applications. These use the standard widget set and standard capabilities. Cross-platform works well.
\nHigh escape-hatch pressure: camera and media processing, background execution, Bluetooth and hardware peripherals, complex custom rendering, deep platform integration (widgets, shortcuts, app extensions), anything real-time.
\nFor high-pressure apps, cross-platform frequently costs more than native, because you pay for the framework and then write the native code anyway.
\nBe honest about which one you are before choosing. Most teams that regret their choice were high-pressure and assessed themselves as low.
\nReact Native. Mature, large ecosystem, native widgets. The new architecture removed the old asynchronous bridge, which was the main performance complaint. Best choice if your team is already React.
\nFlutter. Renders its own widgets, which means true visual consistency and a non-native feel that some users notice and most do not. Excellent performance for custom interfaces. Dart is a real adoption cost for a team that does not know it.
\nKotlin Multiplatform. Share business logic, write native UI. This is the approach I find most defensible: the logic layer \u2014 networking, models, validation, persistence \u2014 is where duplication is most wasteful and least visible to users, and the UI layer is where platform idiom matters most.
\nRequires two UI implementations, which is the point rather than a limitation.
\nWeb technologies in a wrapper. Fastest to build if you have web engineers, and the platform feel is the weakest. Fine for content-heavy applications, poor for anything interaction-heavy.
\nNative. Still correct for a large category, and the category is smaller than native advocates believe.
\nDo you have or can you hire two platform teams?
\nIf yes, native is viable and gives you the best result.
\nIf no \u2014 and for most companies below a certain size the answer is no \u2014 the choice is between cross-platform and shipping on one platform. Framed that way, the decision is usually easy.
\nWhat is your feature velocity?
\nIf you ship a large feature monthly, implementing it twice is a permanent 2\u00d7 cost on your most expensive activity. If you ship quarterly, the duplication matters less.
\nHow much does platform idiom matter to your users?
\nFor a consumer app competing on polish: a lot. For an internal tool: nothing. For a B2B product where the buyer is not the user: less than you think.
\nShare the logic, write the UI natively.
\nThe business logic \u2014 API clients, data models, validation, offline storage, sync, analytics \u2014 is genuinely identical across platforms and duplicating it produces bugs that exist on one platform and not the other, which are the worst bugs to diagnose.
\nThe UI is where platform conventions matter, where users notice, and where the framework abstraction costs the most.
\nThis is more work than full cross-platform and less than full native, and it puts the sharing where the value is.
\nPrototype the hardest thing first.
\nNot the login screen. The thing you are worried about \u2014 the camera flow, the background sync, the complex list, the offline behavior. Build that on your candidate stack, in a week.
\nYou will learn more from that week than from any amount of comparison, and you will learn it while changing your mind is still cheap.
\nThe teams that regret their choice almost always chose based on a comparison article, built the easy part first, and discovered the hard part in month five.
", "image": "https://readme.news/cards/cross-platform-is-a-promise-you-make-to-your-budget.png", "date_published": "2026-06-30T09:00:00Z", "date_modified": "2026-06-30T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "platforms", "architecture", "engineering", "opinion" ] }, { "id": "https://readme.news/retries-a-complete-guide-to-not-making-it-worse/", "url": "https://readme.news/retries-a-complete-guide-to-not-making-it-worse/", "title": "Retries: a complete guide to not making it worse", "summary": "The most common way a small incident becomes a large one is a retry policy written without thinking about aggregate behavior.", "content_html": "Retries are the most commonly implemented and most commonly wrong piece of resilience engineering. A retry policy that seems obviously correct in isolation is frequently the mechanism that turns a brief degradation into a full outage.
\nA downstream service slows down. Every caller times out. Every caller retries.
\nThe downstream now receives double its normal traffic while already struggling. More requests time out. More retries. The load multiplies with every round.
\nThe original problem might have been a five-second blip. The retry storm keeps the service down for twenty minutes, and it stays down after the original cause is resolved because the queued retries are still arriving.
\nThis is metastable failure and retries are its most common cause.
\n1. Only retry idempotent operations.
\nA retried non-idempotent operation charges the card twice. If you need to retry a write, make it idempotent first with an idempotency key.
\n2. Only retry retryable errors.
\nA 400 will be a 400 next time. Retrying it wastes a request and delays the error the caller needs to see.
\n| retry | do not retry |
|---|---|
| connection refused / reset | 400, 401, 403, 404 |
| timeout | 422 |
| 502, 503, 504 | any deterministic validation failure |
429 (respect Retry-After) | 501 |
The one people get wrong: 500 is ambiguous. It might be transient, it might be a deterministic bug. Retrying it once is usually reasonable; retrying it five times is usually pointless.
\n3. Exponential backoff with jitter. Always jitter.
\nWithout jitter, all your clients retry at the same moments, and you have built a synchronized load generator.
\ndef delay(attempt, base=0.1, cap=30):\n return random.uniform(0, min(cap, base * (2 ** attempt)))That is full jitter, and it is the recommended default. It spreads retries across the whole interval, which is what you want. Half jitter \u2014 d/2 + random(0, d/2) \u2014 is a reasonable alternative when you want a guaranteed minimum delay.
The version without jitter is the one everyone writes first and it is the one that causes the storm.
\n4. Cap the attempts and cap the total time.
\nThree attempts, usually. And a total deadline \u2014 if the caller is going to give up after two seconds, retrying at three seconds is pure waste and it is load on a struggling service.
\nPropagate the deadline. If your caller has 500 ms left, your retry budget is 500 ms, not your configured default.
\n5. Do not retry at every layer.
\nThis is the one that produces the shocking numbers. Three attempts at the HTTP client, three in the service wrapper, three in the caller, three at the gateway: 3\u2074 = 81 requests for one logical call.
\nRetry at exactly one layer. Usually the outermost one that has the context to decide. Every other layer fails fast and propagates.
\nAudit this. Most systems that have grown organically retry at three or four layers and nobody knows.
\n6. Use a retry budget.
\nThe refinement that actually prevents storms: cap retries as a fraction of total traffic, not per request.
\nif retries_in_window / requests_in_window > 0.1:\n do_not_retry()When things are healthy, occasional retries are well under the budget and everything works. When things are broken, the budget is exhausted immediately and retries stop entirely \u2014 exactly when they would do the most harm.
\nThis single mechanism converts retries from a failure amplifier into a bounded safety net, and it is not widely implemented.
\n7. Circuit break.
\nWhen a downstream is clearly failing, stop calling it. Fail fast, return a cached or degraded response, and probe occasionally to see if it has recovered.
\nThree states: closed (normal), open (failing fast), half-open (probing). The half-open state must allow only a trickle \u2014 if you send full traffic at a recovering service you will knock it over again.
\nThe other half, which is usually forgotten.
\nSend Retry-After on 429 and 503. Then clients that respect it retry at a time you chose rather than a time they chose.
Shed load rather than queueing it. A request that will time out anyway should be rejected immediately, not queued. Queueing under overload increases latency for everything without increasing throughput, and the requests you eventually serve have often already been abandoned.
\nPrioritize. Under load, serve health checks and critical paths, shed the rest. An unprioritized overload sheds randomly, which means your health checks fail and your orchestrator kills healthy instances.
\nTake your service. Make a downstream dependency return 503 for everything. Watch the request rate at the downstream.
\nIf it goes up by more than a small factor, your retry configuration will cause an outage. You have just not had the trigger yet.
", "image": "https://readme.news/cards/retries-a-complete-guide-to-not-making-it-worse.png", "date_published": "2026-06-29T09:00:00Z", "date_modified": "2026-06-29T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "reliability", "architecture", "engineering" ] }, { "id": "https://readme.news/half-of-2026-in-one-page/", "url": "https://readme.news/half-of-2026-in-one-page/", "title": "Half of 2026, in one page", "summary": "Six months of regulation, agent tooling that mostly did not fix the bottleneck, and a web platform that quietly finished.", "content_html": "The first half, compressed.
\nRegulation became an engineering concern. The EU AI Act's high-risk obligations arrive in August; the Cyber Resilience Act's reporting obligations in September; post-quantum migration guidance firmed up across national agencies. For the first time in a while, compliance is showing up on engineering roadmaps rather than only in legal reviews.
\nPower stayed the binding constraint on compute. Interconnection queues, transformer lead times, and utility rate cases determine how fast capacity arrives. Every roadmap presented this year is downstream of a construction schedule.
\nThe verification bottleneck did not move. Generation capability kept improving. Review capacity did not. The tooling investment continued to go into producing more code rather than into checking it, which is the wrong end of the problem and which I predicted would correct by now. It has not.
\nGeneration got roughly two orders of magnitude cheaper. Verification got no cheaper at all.
\nEverything else \u2014 the review queue, the junior hiring contraction, the sudden importance of tests and type systems, the unease people cannot name about agent-written code \u2014 is a consequence of that asymmetry, and six months of watching it has not changed my read.
\nThe skill that matters is the ability to look at plausible output and tell whether it is right. That skill is built by doing the work, and the work is being automated.
\nNobody has solved this. Most of the discussion still treats it as either a non-problem or an apocalypse, and it is neither. It is an engineering and training problem with no established answer yet.
\nIn January I made five predictions. Three months in I graded one as wrong. At the halfway point:
\nReview tooling as a major funded category \u2014 still not happening at the scale I predicted. Funding continues to flow to generation. I was wrong about the market, and possibly wrong about the timing rather than the direction.
\nRepository-level agent configuration as standard practice \u2014 correct, and it happened faster than I expected. This is now table stakes for a well-maintained repository.
\nPower as a developer-visible line item \u2014 correct and accelerating. Regional compute price differentials are now large enough to affect architecture.
\nSupply chain controls with real teeth \u2014 in progress. The registry-level changes are landing and the migration is as painful as expected.
\n\"Senior engineer\" redefining around judgment \u2014 correct, uncomfortable, and nobody has a good answer for how the next generation acquires it.
\nBack on Monday.
", "image": "https://readme.news/cards/half-of-2026-in-one-page.png", "date_published": "2026-06-26T09:00:00Z", "date_modified": "2026-06-26T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "retrospective" ] }, { "id": "https://readme.news/configuration-is-the-most-under-designed-part-of-your-system/", "url": "https://readme.news/configuration-is-the-most-under-designed-part-of-your-system/", "title": "Configuration is the most under-designed part of your system", "summary": "It has no type system, no tests, no review, and it causes a disproportionate share of outages.", "content_html": "Look at the last ten significant outages you can remember reading about. A disproportionate number were caused by configuration, not code.
\nA config file with a wrong value. A feature flag flipped. A generated file that doubled in size. A DNS record. A permission change.
\nConfiguration receives a fraction of the engineering rigor that code does, and it has comparable power to break things.
\nNo type system. A YAML file will happily contain timeout: 30s where the code expects an integer, or enabled: \"false\" which is a truthy string.
No tests. Nobody writes a test for their config.
\nNo review, or perfunctory review. A config change is \"just a value\" and gets approved in ten seconds.
\nDeployed differently from code. Frequently faster, frequently without staging, frequently without a canary. The safety mechanisms built for code deployment routinely do not cover config.
\nEnvironment drift. Staging and production differ in ways nobody has enumerated, so staging validates a configuration that is not production's.
\nNo rollback story. \"What was this value before\" is often unanswerable.
\n1. Parse and validate at startup, not at use.
\nThe worst failure mode is a config error that manifests three hours later when a rarely-used code path reads a malformed value.
\nValidate everything at boot. Fail loudly and immediately if anything is wrong.
\nclass Settings(BaseModel):\n database_url: PostgresDsn\n timeout_ms: int = Field(gt=0, le=60_000)\n max_connections: int = Field(ge=1, le=1000)\n feature_new_checkout: bool = False\n\nsettings = Settings(**load_config()) # raises at startup, with a clear messageAn application that will not start is far better than one that starts and behaves wrongly.
\n2. Types, with a schema.
\nWhatever your language, there is a library that turns untyped config into a validated typed object. Use it. This eliminates the entire category of string-that-should-be-a-number bugs.
\n3. Config changes go through the same pipeline as code.
\nVersion control. Review. Staging. Canary. Rollback.
\nThe argument against is that config changes need to be fast, especially for incident response. That is a real need and the answer is a small explicitly-defined set of emergency levers \u2014 kill switches, rate limits \u2014 with a fast path, and everything else on the normal pipeline.
\nNot \"all config is fast\" because that is how config takes down your service.
\n4. Validate generated configuration.
\nIf a config file is produced by a program, that program can be wrong. A size check against the previous version, a schema check, a sanity check on record count.
\nThis is the specific failure that has caused several high-profile outages: a generated file changed unexpectedly and propagated globally before anyone looked at it.
\n5. Make the environment explicit and diff it.
\nYou should be able to answer \"how does staging differ from production\" with a command. If you cannot, staging is not validating production.
\nconfig-diff staging production6. Log the effective configuration at startup.
\nNot the file \u2014 the resolved values after defaults, overrides, and environment variables are applied. Redact secrets. This is the single most useful thing for debugging \"it works on my machine,\" because the effective config is frequently not what anyone thinks it is.
\n7. Feature flags need the same discipline as code.
\nOwner. Expiry. Test both branches. Log the flag state on every event. A flag flip is a production change and should be treated as one.
\nMost systems end up with layered configuration and the layering should be explicit and simple:
\ndefaults in code\n \u2190 config file\n \u2190 environment variables\n \u2190 command line flagsLater overrides earlier. Log which layer each effective value came from when debugging.
\nKeep the layers few. Systems with six overlapping sources of configuration \u2014 defaults, file, environment, a service, a database table, a flag system \u2014 produce values nobody can trace. Each layer you add makes \"why is this value what it is\" harder to answer.
\nNot the same thing as configuration and should not be in the same place.
\nConfiguration is the input to your program that is most likely to be wrong and least likely to be checked.
\nEvery rigor you apply to code \u2014 types, validation, tests, review, staged rollout, rollback \u2014 applies to configuration, and applying it costs a day of setup.
\nThe reason it does not happen is that config does not feel like code. It is data, and data feels safe.
\nIt is not data. It is the arguments to your program, and passing wrong arguments to a program is how programs go wrong.
", "image": "https://readme.news/cards/configuration-is-the-most-under-designed-part-of-your-system.png", "date_published": "2026-06-24T09:00:00Z", "date_modified": "2026-06-24T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "engineering", "reliability", "architecture", "craft" ] }, { "id": "https://readme.news/randomness-and-the-bugs-you-cannot-reproduce/", "url": "https://readme.news/randomness-and-the-bugs-you-cannot-reproduce/", "title": "Randomness, and the bugs you cannot reproduce", "summary": "The bug that happens once a week and never in staging. A systematic approach to the class of problem everyone handles badly.", "content_html": "The worst bugs are the ones that happen sometimes. They resist the standard debugging loop entirely, because the loop requires reproduction and reproduction is exactly what you do not have.
\nMost engineers approach these by staring at code and hoping. There is a better method.
\nThere is a finite list. Work through it.
\nConcurrency. Two things running at once with insufficient ordering. The largest category by far. Includes: unsynchronized shared state, check-then-act races, lost updates, and the classic where two requests both check \"does this exist\" and both create it.
\nTime. Anything that depends on wall clock: timeouts, expiry, scheduled work, date boundaries. These fail at midnight, at month end, on the DST transition, on leap day, and when NTP adjusts the clock backward.
\nOrdering. Hash map iteration order, filesystem directory order, unordered message delivery, parallel test execution. Code that accidentally depends on an order that is not guaranteed works until it does not.
\nExternal state. A cache that is sometimes warm. A connection that is sometimes pooled. A DNS response that sometimes returns a different IP. A downstream that is sometimes slow.
\nResource exhaustion. Memory pressure changes GC timing, which changes interleaving. Connection pool exhaustion changes behavior under load. File descriptor limits. These make other latent bugs appear.
\nActual randomness. UUIDs, load balancing, sampling, retries with jitter, partitioning by hash.
\nUninitialized memory or undefined behavior, in languages that permit it.
\n1. Instrument before you theorize.
\nYou cannot reproduce it, so you must capture it. Add logging around the suspicious area \u2014 not \"entering function,\" but the actual values, the timing, the thread or task identity, the state.
\nThe instinct is to avoid adding logging to production. Add it. A bug you cannot reproduce is a bug you must observe in the wild, and that requires observation.
\n2. Find the correlation.
\nYou have occurrences. What do they have in common?
\nThis is why wide structured events matter. If every request logs its context, the correlation is a query. Without it, you are guessing.
\n3. Make it more likely.
\nOnce you have a hypothesis, try to increase the failure rate:
\nMaking a rare bug common is the single most effective technique in this whole category, and it is underused because it feels like making things worse.
\n4. Add an assertion.
\nIf you believe an invariant holds, assert it. In production, with an alert.
\nassert order.total == sum(i.price * i.qty for i in order.items), \\\n f\"order {order.id} total mismatch: {order.total}\"You will find out that it does not hold, and you will find out with the context attached rather than downstream where the symptom appears.
\nThe window between \"the invariant broke\" and \"someone noticed the symptom\" is where the information lives, and assertions collapse it to zero.
\n5. Bisect it.
\nIf it started recently, git bisect still works on statistical failures \u2014 you just need a test script that runs the operation many times and reports failure if the rate exceeds a threshold. Slower than a deterministic bisect and far better than reading diffs.
Make things deterministic where you can. Inject the clock rather than calling it. Inject the random source with a seed. Sort collections before iterating where order matters. Deterministic systems have bugs you can reproduce, which means bugs you can fix.
\nRun tests in randomized order, with a printed seed. If a test only passes in a specific order, it has a hidden dependency, and that dependency is a bug in your production code more often than people assume.
\nUse your language's race detector. Go's -race, thread sanitizer for C and C++, Java's concurrency tooling. These find real races that have never yet manifested. Run them in CI.
Prefer immutability. A value that cannot change cannot be changed concurrently. This eliminates the largest category of nondeterminism structurally.
\nLog the identifiers. Trace ID, request ID, tenant. Nine tenths of the difficulty in these investigations is being unable to correlate events you already recorded.
\nSome of these take weeks. A concurrency bug that occurs once per million requests in a system with a subtle ordering dependency is genuinely difficult, and no method makes it easy.
\nThe method makes it tractable: instead of staring and hoping, you have a list of candidate sources, a way to gather evidence, and a way to raise the failure rate until it is reproducible.
\nThat is the difference between a bug you eventually fix and one that sits in the backlog for two years labeled \"cannot reproduce.\"
", "image": "https://readme.news/cards/randomness-and-the-bugs-you-cannot-reproduce.png", "date_published": "2026-06-22T09:00:00Z", "date_modified": "2026-06-22T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "craft", "engineering", "reliability" ] }, { "id": "https://readme.news/what-a-staff-engineer-does-all-day/", "url": "https://readme.news/what-a-staff-engineer-does-all-day/", "title": "What a staff engineer does all day", "summary": "The role is genuinely ambiguous and that ambiguity is load-bearing. An attempt at a concrete description.", "content_html": "\"Staff engineer\" is the most poorly-defined common title in the industry. It means different things at different companies, and even within one company two staff engineers may do almost nothing in common.
\nHere is an attempt at what the role actually is, based on watching people do it well.
\nNot \"a senior engineer who has been there longer.\" Time in seat produces a senior engineer with more context, which is valuable and is not this.
\nNot \"the best coder.\" The highest-output individual contributor is a valuable role and it is a different one. Staff engineers frequently write less code than seniors.
\nNot \"a manager who did not want to manage.\" The scope is comparable to a manager's; the mechanism is entirely different.
\nThe concise version: a staff engineer is responsible for the technical success of work that spans more than one team, without having authority over those teams.
\nEverything distinctive about the role follows from \"without authority.\" You cannot assign work. You cannot approve headcount. You cannot make a decision stick by deciding it. Every outcome has to be achieved through information, credibility, and persuasion.
\nFinding the problem nobody owns. The most common form of staff-level impact. Every organization has problems that fall between teams: the shared library nobody maintains, the integration that fails and each side thinks is the other's, the performance issue that is caused by the interaction of three services.
\nNobody owns these, so nobody fixes them, so they persist for years. Identifying one, proving it matters, and getting it fixed is a substantial contribution and it usually requires a person who is not on any of the teams involved.
\nMaking a decision that spans teams. Which of three approaches, when each team has a preference and none of them can see the whole picture. This is where the \"no authority\" constraint bites hardest: the decision has to be made in a way that the people affected accept, which means the reasoning has to be visible and the objections have to be genuinely addressed.
\nWriting the document that ends the argument. A recurring disagreement that resurfaces every quarter because nobody wrote down the resolution. One good document \u2014 with the options, the trade-offs, the decision, and the reasoning \u2014 can end a multi-year debate.
\nBeing the person who read the whole system. Most engineers know their team's code. Somebody needs to know how it all fits together, including the parts nobody has touched in three years. That knowledge is what makes the cross-cutting problems visible.
\nRaising the floor. Not by writing better code. By making the good pattern easy: a library that removes a class of bug, a template that encodes the right defaults, a lint rule that prevents the mistake, documentation that means nobody has to ask. Impact through leverage rather than output.
\nMentoring, specifically on judgment. Not \"how do I use this API.\" \"Should we build this at all,\" \"how do I disagree with my manager about a technical decision,\" \"how do I tell if this design will be a problem in a year.\"
\nSaying no with a reason, at a level where it lands. Frequently the most valuable thing a staff engineer does, and the reason it requires seniority is that the no has to come with an alternative and with credibility behind it.
\nA real week looks roughly like:
\nThe proportion of coding is the part that surprises people moving into the role, and the discomfort of not shipping visible code is the most common reason people bounce out of it.
\nDo things get decided? Not \"do they have opinions.\" Do arguments they are involved in reach a resolution that holds.
\nDo other engineers get better? Look at the people around them over a year.
\nDo they work on things nobody asked them to? The highest-value staff work is usually self-directed, because if it were obvious and assigned it would already have an owner.
\nAre they trusted by people who disagree with them? This is the real test. Someone who is only trusted by people who already agree has influence, not credibility.
\nBecoming an architecture astronaut. Designs, diagrams, opinions, no contact with running code. The credibility runs out within about a year, and it is very hard to get back.
\nBecoming a very expensive senior engineer. Doing excellent work with single-team scope. Comfortable, valuable, and not the job.
\nSpreading too thin. Involved in twelve things, effective in none. The scope is tempting and the constraint is real: two or three significant efforts at a time is the realistic maximum.
\nLosing the ability to build. The role requires enough hands-on work to stay credible and calibrated. An engineer who has not shipped in a year is guessing.
", "image": "https://readme.news/cards/what-a-staff-engineer-does-all-day.png", "date_published": "2026-06-19T09:00:00Z", "date_modified": "2026-06-19T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "engineering-management", "careers", "craft", "opinion" ] }, { "id": "https://readme.news/licenses-a-practical-guide-for-people-who-ship/", "url": "https://readme.news/licenses-a-practical-guide-for-people-who-ship/", "title": "Licenses: a practical guide for people who ship", "summary": "Not legal advice. A working engineer's map of what the common licenses actually require you to do.", "content_html": "Most engineers can name three licenses and could not tell you what any of them require. That is a problem when your product ships hundreds of dependencies and somebody in legal eventually asks.
\nThis is not legal advice. It is the working map.
\nMIT, BSD (2- and 3-clause), ISC, Apache 2.0.
\nWhat you must do: include the license text and the copyright notice with your distribution. That is essentially it.
\nWhat you may do: everything. Use it commercially, modify it, ship it in a closed product, sublicense it.
\nApache 2.0 additionally: grants patent rights explicitly, and terminates those rights if you sue a contributor for patent infringement over the software. It also requires you to state significant changes you made.
\nThat patent grant is why Apache 2.0 is preferred over MIT by legal departments at larger companies. MIT is silent on patents, which is not the same as safe.
\nIn practice: include a NOTICES file or an attribution page listing your dependencies and their license texts. Generate it from your dependency tree. This is the entire compliance obligation for the permissive family and most products do not do it.
LGPL, MPL 2.0, EPL.
\nThe rule: modifications to the licensed files must be released under the same license. Your own code that merely uses the library does not have to be.
\nMPL 2.0 is file-based, which is the cleanest formulation: if you modify a file that came under MPL, that file stays MPL. Everything else is yours.
\nLGPL is linking-based and the details matter. Dynamic linking is generally considered fine \u2014 the user must be able to replace the library. Static linking requires either providing object files so the user can relink, or releasing under compatible terms.
\nFor most modern language ecosystems, \"dynamic linking\" does not map cleanly onto how code is actually distributed, and this is a genuine gray area. If you statically link LGPL code into a distributed binary, that is worth a real legal conversation.
\nIn practice: MPL and EPL are unproblematic for most commercial use. LGPL is usually fine and deserves attention if you are shipping a compiled binary rather than running a service.
\nGPL v2, GPL v3.
\nThe rule: if you distribute a work based on GPL code, the whole work must be under the GPL, and you must provide source.
\nThe key question is what \"distribute\" means. Running GPL software on your server and letting users access it over a network is not distribution. This is why a very large amount of GPL software runs inside commercial SaaS products without obligation \u2014 Linux being the obvious example.
\nGPL v3 additionally: anti-tivoization (you must let users install modified versions on hardware you ship), and explicit patent provisions.
\nIn practice: GPL dependencies in a hosted service are usually fine. GPL dependencies in software you ship to customers make your software GPL, which is usually not what you intended.
\nKnow which of your dependencies are GPL. Most people do not.
\nAGPL v3.
\nThe rule: GPL, plus \u2014 if users interact with the software over a network, they must be able to get the source, including your modifications.
\nThis closes the \"SaaS loophole\" deliberately. It is why many companies have a blanket policy prohibiting AGPL dependencies: the obligation is triggered by normal SaaS operation, and determining exactly how much of your system counts as \"the software\" is a question nobody wants to litigate.
\nIn practice: if your company has a license policy, AGPL is almost certainly on the prohibited list. Check before you add the dependency, not after.
\nBUSL, SSPL, Elastic License, various \"fair source\" licenses.
\nThese are not open source licenses. They restrict commercial use, usually prohibiting offering the software as a service that competes with the licensor.
\nThey exist because hyperscalers built managed services on open source projects without contributing back, and the projects had no recourse. That grievance is real.
\nIn practice: read the actual text, every time. They differ substantially. Many convert to an open source license after a delay \u2014 BUSL typically becomes Apache 2.0 after four years \u2014 which means the version you are using today may become permissive before you care.
\nThe specific question to answer: does your use case compete with the licensor's commercial offering? Usually the answer is obviously no and you are fine.
\n1. Generate an SBOM in CI. Every build, listing every dependency and its license. Tooling for this is mature in every major ecosystem.
\n2. Fail the build on prohibited licenses. Have a policy \u2014 usually: permissive fine, weak copyleft fine, GPL depends on whether you distribute, AGPL and source-available require review.
\n3. Generate the attribution file automatically. This is the actual compliance obligation for the licenses you are most likely to use, and generating it is a build step, not a project.
\n4. Check when you add, not when you ship. The cost of removing a dependency after it is embedded is enormous. The cost of checking at npm add time is zero.
License compatibility is not transitive intuition. You can use GPL code in a GPL project. You cannot use GPL code in an Apache-licensed library that others will embed in proprietary software \u2014 you have made your library effectively GPL, and your users will find out later.
\nIf you publish a library, its license must be compatible with every dependency it pulls in. Check that specifically, because it is the mistake that is expensive to discover after adoption.
", "image": "https://readme.news/cards/licenses-a-practical-guide-for-people-who-ship.png", "date_published": "2026-06-17T09:00:00Z", "date_modified": "2026-06-17T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "open-source", "policy", "engineering" ] }, { "id": "https://readme.news/distributed-tracing-that-people-actually-use/", "url": "https://readme.news/distributed-tracing-that-people-actually-use/", "title": "Distributed tracing that people actually use", "summary": "Most tracing deployments produce beautiful waterfalls nobody opens. The difference is three implementation details.", "content_html": "Distributed tracing is the right answer to \"what happened to this request across nine services.\" Most implementations produce a system that is technically correct and that nobody opens during an incident.
\nThree details separate the two outcomes.
\nA trace is only useful if you can find it. Which means the trace ID must appear:
\nThe workflow that makes tracing valuable: customer reports a problem \u2192 support gets the trace ID \u2192 engineer opens exactly that request and sees everything.
\nWithout the ID in those places, the workflow is: customer reports a problem \u2192 engineer tries to guess which of eight million traces it was.
\nThat single difference determines whether the investment pays off, and it is a plumbing problem rather than a tracing problem.
\nA span that says \"database query, 45 ms\" is nearly useless. Which query? Which table? How many rows? Was it cached?
\nwith tracer.start_as_current_span(\"db.query\") as span:\n span.set_attribute(\"db.system\", \"postgresql\")\n span.set_attribute(\"db.operation\", \"select\")\n span.set_attribute(\"db.table\", \"orders\")\n span.set_attribute(\"db.rows_returned\", len(rows))\n span.set_attribute(\"app.tenant_id\", tenant)\n span.set_attribute(\"app.cache_hit\", cached)The attributes are what let you ask questions the waterfall cannot answer: is this slow for one tenant, is it slow when the cache misses, is it slow when the result set is large.
\nPut your business identifiers on the root span. Tenant, user, plan tier, feature flags, client version. Then you can filter traces by them, which is how you find the pattern rather than the instance.
\nHigh cardinality is correct here. This is not a metrics system.
\nFull sampling is expensive and mostly wasteful \u2014 the successful, fast, boring requests are identical to each other.
\nTail-based sampling makes the decision after the trace completes, when you know whether it was interesting:
\nThat last one is worth building explicitly. A flag that says \"capture everything for this tenant for the next hour\" turns an unreproducible customer report into a solvable problem, and it is the single most useful debugging feature you can add to a multi-tenant system.
\nInstrumenting everything. Auto-instrumentation gives you spans for every framework operation, most of which are noise. A trace with four hundred spans is not more informative than one with twenty; it is less, because the signal is buried.
\nInstrument the boundaries \u2014 service calls, database queries, external APIs, queue operations \u2014 and add manual spans only for genuinely expensive internal operations.
\nPerfect propagation. You will have services that drop the context: a message queue without header support, a third-party integration, a legacy component. Do not block the rollout on 100% coverage. A trace with a gap is still much better than no trace.
\nBuilding dashboards from traces. Traces answer \"what happened to this request.\" Metrics answer \"what is happening to all requests.\" Using traces for aggregate views is expensive and slow. Use both, for what each is good at.
\nNot \"is the system slow\" \u2014 metrics tell you that, cheaper.
\nNot \"what error happened\" \u2014 logs tell you that.
\n**\"Why was this specific request slow, and what was different about it?\"**
\nThat is the question. It is the question you have during an incident, when one customer is affected and the aggregate metrics look fine. And it is unanswerable without tracing.
\nIf your tracing setup does not make that question easy to answer in under a minute, the setup is the problem, not the concept.
\nIf you are starting from nothing:
\nThat is a week of work and it covers most of the value. Everything beyond it is refinement.
", "image": "https://readme.news/cards/distributed-tracing-that-people-actually-use.png", "date_published": "2026-06-15T09:00:00Z", "date_modified": "2026-06-15T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "reliability", "engineering", "craft" ] }, { "id": "https://readme.news/refactoring-under-an-agent/", "url": "https://readme.news/refactoring-under-an-agent/", "title": "Refactoring under an agent", "summary": "Large mechanical refactors are the clearest win available from coding agents. Here is the process that keeps them safe.", "content_html": "Large mechanical refactors \u2014 rename this concept across four hundred files, migrate every call site to a new API, convert a pattern used everywhere \u2014 are the single clearest win available from coding agents.
\nThey are also where an unattended agent can do the most damage quietly. Here is the process that has worked.
\nMechanical refactors have exactly the properties agents are good at:
\nAnd they have the property that makes human refactoring risky: the scale defeats attention. A person converting four hundred call sites will be careful for the first fifty.
\n1. Do ten by hand first.
\nBefore writing any instruction, do a representative sample yourself. You will discover:
\nThis is the step people skip and it is the one that determines whether the whole thing works. You cannot specify a transformation you have not performed.
\n2. Write the transformation down precisely.
\nNot \"modernize the error handling.\" The exact before and after, with the exceptions named:
\nReplace every call of the form:\n result, err := doThing(x)\n if err != nil { return nil, err }\nwith:\n result, err := doThing(x)\n if err != nil { return nil, fmt.Errorf(\"doing thing for %s: %w\", x.ID, err) }\n\nExceptions:\n- Do not change anything in internal/legacy/ (frozen).\n- Do not change error handling inside deferred functions.\n- If the error is already wrapped, leave it.3. Establish the safety net before you start.
\n4. Batch it.
\nNot four hundred files in one change. Twenty to fifty files per commit, grouped by module.
\nThis matters for two reasons: a reviewable diff size, and the ability to bisect. If something is wrong, you want to know which batch introduced it.
\n5. Review the first batch line by line.
\nEvery line. This is where you catch the systematic error, and a systematic error caught in batch one costs twenty minutes while the same error caught in batch twenty costs a day.
\n6. Spot-check subsequent batches, review the anomalies.
\nOnce the pattern is verified, review by sampling \u2014 but read every diff that looks different from the pattern. Agent output that deviates from the established shape is where the interesting failures are.
\n7. Verify mechanically at the end.
\n# nothing left in the old form\nrg 'return nil, err$' --type go | rg -v 'internal/legacy'If the transformation is complete, the old pattern should not exist outside the exceptions. This catches the files that were silently skipped, which is a real failure mode.
\nThe agent \"improves\" things you did not ask about. It renames a variable while fixing the error handling. Individually reasonable, collectively it makes the diff unreviewable because you can no longer scan for the pattern.
\nInstruct explicitly: change only what was specified, nothing else.
\nSemantic drift across batches. Batch one wraps errors one way, batch fifteen does it slightly differently, because the context is different and the model made a different reasonable choice.
\nFix: put the exact target form in the instruction file, with examples, and check for consistency at the end with a grep.
\nTests pass and behavior changed. The most dangerous case. Your tests did not cover the path that broke.
\nThis is why the mechanical verification in step seven matters. It checks the transformation, not the behavior, and it catches things tests do not.
\nSilent skips. The agent processes 380 of 400 files and reports success. The twenty it skipped are the ones with unusual structure, which are the ones most likely to matter.
\nAlways count. Always verify the remainder is empty.
\nFor this class of work the leverage is real and large. A refactor that would have been a week of tedium \u2014 and would therefore never have happened, which is why the codebase has the problem \u2014 becomes an afternoon.
\nThat last part is the underrated benefit. The refactors that get done are the ones that are cheap enough to do. Lowering the cost means more of them happen, and a codebase where the cross-cutting cleanup actually gets done is a meaningfully better codebase.
\nJust count the files at the end.
", "image": "https://readme.news/cards/refactoring-under-an-agent.png", "date_published": "2026-06-12T09:00:00Z", "date_modified": "2026-06-12T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "ai", "coding-agents", "craft" ] }, { "id": "https://readme.news/the-world-cup-is-the-largest-load-test-ever-run/", "url": "https://readme.news/the-world-cup-is-the-largest-load-test-ever-run/", "title": "The World Cup is the largest load test ever run", "summary": "A month of synchronized global demand across three countries, sixteen cities, and every streaming platform at once.", "content_html": "The tournament starts tomorrow: forty-eight teams, three host countries, sixteen venues, and a match schedule designed so that a very large fraction of the planet is watching the same thing at the same moment, repeatedly, for a month.
\nFrom an engineering perspective this is the most demanding recurring event in consumer computing, and almost nothing about how it works gets written up.
\nSynchronized, not smooth. Streaming traffic for a scheduled match is a step function. Millions of concurrent sessions establish within a two-minute window around kickoff, and the ones that fail to establish are a product failure with no recovery \u2014 the user missed the start.
\nMulti-peak within a session. A goal produces a spike in social traffic, in betting platforms, in messaging, in news sites, and in the streams of people switching from another match. These arrive within seconds of each other and are correlated across completely unrelated companies.
\nCorrelated across the industry. This is the part that is genuinely unusual. Every CDN, every mobile network, every payment processor, and every messaging platform experiences the peak simultaneously. There is no borrowing capacity from a quiet neighbor, because there is no quiet neighbor.
\nMulti-region with different profiles. Matches in three countries across several time zones means the traffic profile shifts across the tournament in ways that capacity planning must anticipate.
\nPayment processing at scale. Betting and merchandise platforms see enormous transaction spikes at specific moments. Payment processors are a shared dependency and their capacity is a hard constraint nobody downstream controls.
\nMobile networks in venue areas. Sixty thousand people in one place, all trying to upload video. This is a well-understood problem with an expensive solution (temporary cell capacity) and it still degrades.
\nAuthentication systems. Everyone logs in at once. Login is frequently the least scaled part of a streaming stack because it is not on the hot path during normal operation.
\nThe thundering herd on recovery. Something fails, comes back, and every client reconnects simultaneously \u2014 causing a second failure. Retry jitter is the single most important line of code in a system like this and it is routinely absent.
\nAd insertion. Server-side ad insertion personalizes streams per viewer while keeping segments cacheable. It is a hard constraint and it is a disproportionate source of live-stream incidents.
\nPre-scale, do not autoscale. Autoscaling responds to load after it arrives. Instance startup is measured in tens of seconds; the demand arrives in one. For a known event at a known time, capacity is provisioned in advance and the autoscaler is a safety net, not the mechanism.
\nLoad shed by feature, not by user. When capacity is short, disable the recommendations, the comments, the statistics overlay \u2014 keep the video. A degraded experience for everyone beats a perfect experience for 80% and nothing for the rest.
\nMulti-CDN with active steering. Not for capacity alone \u2014 for the fact that any single CDN will have a bad region on a given day. A client-side or DNS-level steering layer that measures real performance and shifts traffic is the difference between a degraded minute and an outage.
\nRehearse. Group stage matches are the rehearsal for the knockout rounds. The teams that treat early matches as production load tests, with instrumentation and a retrospective after each one, are the ones that survive the final.
\nA war room with authority. Not a monitoring dashboard \u2014 a room with the people who can make decisions, including the decision to turn features off, without an approval chain.
\nThe interesting property here is demand you cannot smooth, shed, or refuse.
\nMost systems get to spread load over time, queue it, or degrade gracefully by making users wait. None of those work when the product is a live event \u2014 a queue means the user misses the goal.
\nWhat is left is: provision for peak, make every layer redundant, degrade by feature rather than by user, and rehearse.
\nThat is expensive, unglamorous, and the only thing that works. It is worth remembering the next time someone proposes autoscaling as the answer to a spike that arrives faster than a machine can boot.
\nSome capacity you have to already own.
", "image": "https://readme.news/cards/the-world-cup-is-the-largest-load-test-ever-run.png", "date_published": "2026-06-10T09:00:00Z", "date_modified": "2026-06-10T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "infrastructure", "reliability" ] }, { "id": "https://readme.news/wwdc-2026-and-the-platform-that-keeps-its-own-counsel/", "url": "https://readme.news/wwdc-2026-and-the-platform-that-keeps-its-own-counsel/", "title": "WWDC 2026 and the platform that keeps its own counsel", "summary": "New OS versions, more on-device model surface, and a developer relationship that remains complicated.", "content_html": "Apple's developer conference ran this week. The pattern of the last few years continues: strong on-device capability, tight platform integration, and a set of platform policy questions that are being settled in courtrooms rather than on stage.
\nApple's position has been consistent and, I think, correct for their situation:
\nThat is not going to win a benchmark comparison and it was never trying to. It is going to win on the axis where Apple competes, which is a coherent product where the default behavior is the one most users want.
\nFor developers, the practical implication has not changed: design features so the small model handles the common case.
\nConcretely \u2014 the 90% that the on-device model can do is free, instant, and works on a plane. The 10% that needs escalation costs money and needs a network. Getting that split right is the engineering, and it is a different skill from prompt design.
\nSwift continues its expansion beyond Apple platforms \u2014 server-side, embedded, WebAssembly, cross-platform tooling. The concurrency model's strict checking continues to produce a language that catches data races at compile time, which is a genuinely valuable property that the migration cost has made contentious.
\nThe honest state: strict concurrency checking is correct and it is a real migration burden for existing codebases, and the error messages when you get it wrong are still harder to act on than they should be.
\nSwift is a better language than it gets credit for outside the Apple ecosystem and its adoption outside that ecosystem remains limited, mostly for reasons of tooling gravity rather than language quality.
\nThe regulatory pressure on app distribution and payments continues, differently in different jurisdictions, with the result that the rules now vary by region in ways that are genuinely confusing for developers to comply with.
\nI do not have a clean position on the substance. What I will say is the practical part:
\nIf you ship an app with any monetization, you now have jurisdiction-specific compliance work, and the rules are still moving. Budget for it, watch the changes, and do not assume a policy you read last year is current.
\nThe broader observation, which is not about Apple specifically: the era of one global set of platform rules is over. Every major platform now operates under different obligations in the EU, the US, and several other markets, and that fragmentation is going to increase rather than resolve.
\nFiltering out the keynote, the items that will matter to a working developer:
\nAnything that reduces app size or launch time. These are the metrics that correlate with retention and they get the least stage time.
\nTesting and debugging improvements in the toolchain. Consistently the most valuable and least covered part of every WWDC.
\nDeprecation notices. The most important announcements at any platform conference are the ones about what is going away, and they are never in the keynote. Read the release notes.
\nApple ships coherent, well-integrated platforms with genuinely good on-device capability and a privacy posture that is a real product differentiator rather than only marketing.
\nIt also operates the developer relationship with less flexibility than any comparable platform, and the regulatory environment is the mechanism by which that is being adjusted rather than any change of heart.
\nBoth of those have been true for a decade and neither is changing this year.
", "image": "https://readme.news/cards/wwdc-2026-and-the-platform-that-keeps-its-own-counsel.png", "date_published": "2026-06-08T09:00:00Z", "date_modified": "2026-06-08T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "platforms", "languages", "ai" ] }, { "id": "https://readme.news/static-analysis-is-finally-worth-the-false-positives/", "url": "https://readme.news/static-analysis-is-finally-worth-the-false-positives/", "title": "Static analysis is finally worth the false positives", "summary": "The tools got dramatically better while everyone was ignoring them because of a bad experience in 2015.", "content_html": "A lot of engineers formed their opinion of static analysis from a tool that produced four thousand warnings on first run, 95% of which were noise, and got disabled within a month.
\nThat was an accurate assessment of the tools at the time. The tools are substantially different now and the assessment has not updated.
\nFlow-sensitive analysis became standard. Older linters matched patterns in the syntax tree. Modern analyzers track values through the control flow graph, which means they can tell that a variable was checked for null on line 12 and therefore is not null on line 40. That single capability eliminates the largest source of false positives.
\nLanguage servers made it interactive. A warning in your editor while you type is a different product from a report generated in CI. You fix it in context, in seconds, instead of triaging a list a week later.
\nType systems absorbed much of it. A lot of what static analysis used to catch is now caught by the compiler in a typed language, for free, with no false positives at all.
\nThe defaults got sane. Modern tools ship with a curated recommended set rather than everything enabled. clippy, ruff, biome, staticcheck and their peers are opinionated about what is worth reporting.
They got fast. Analyzers written in compiled languages run over a large codebase in seconds. Speed matters more than people credit \u2014 a check that takes two minutes gets run in CI, and a check that takes two seconds gets run on every save, which is where it actually changes behavior.
\nA fast linter with a good default set, on save, in the editor. ruff for Python, clippy for Rust, biome or eslint for JavaScript, staticcheck for Go.
A typechecker in strict mode, in CI. This is the highest-value item on the list for a gradually-typed language, and the non-strict modes permit exactly the holes that make the guarantees unreliable.
\nA security-focused analyzer if you handle untrusted input. Taint tracking \u2014 does data from a request reach a SQL query, a shell command, or a template without sanitization \u2014 is the specific capability worth having, and it is a genuinely different analysis from ordinary linting.
\nA dependency scanner, ranked by reachability if your tooling supports it. A critical CVE in code you never call is lower priority than a medium in your request path, and a scanner that cannot tell you which is which produces a queue nobody reads.
\nTurning on a full rule set against an existing codebase produces thousands of warnings and gets the tool disabled. The sequence that works:
\n1. Run it in report-only mode. Get the number. Do not fix anything yet.
\n2. Enable a small subset that has near-zero false positives and fix those. Usually: unused variables, unreachable code, obviously wrong comparisons, missing awaits. Twenty rules, not four hundred.
\n3. Make it blocking for new and changed code only. Most tools support this, either natively or through a diff-aware wrapper. This is the key move \u2014 the existing violations do not block anyone, and the codebase stops getting worse immediately.
\n4. Burn down the backlog opportunistically. When you touch a file, fix its warnings. No cleanup sprint, no dedicated project.
\n5. Add rules gradually, one at a time, each with a burn-down.
\nSteps three and four are where most adoptions succeed or fail. A tool that blocks the whole team on a pre-existing backlog gets turned off; one that only blocks new violations is uncontroversial.
\nSome checks are genuinely contested and you should decide deliberately rather than accepting the default:
\nCyclomatic complexity limits. Sometimes a function is legitimately complex because the domain is. A hard limit produces artificially split functions that are harder to read, not easier.
\nLine length. Real disagreement, formatter should handle it, not worth a rule.
\nNaming conventions. Worth enforcing, and pick your convention rather than the tool's default if they differ.
\nAnything with more than a few percent false positives. A rule that is wrong one time in ten trains people to ignore it, and that habit generalizes to the rules that are right.
\nStatic analysis finds a specific class of bug: local, syntactic, pattern-matchable. It does not find logic errors, wrong business rules, race conditions in most cases, or performance problems.
\nIt is not a substitute for tests, review, or thought. It is a way to spend zero human attention on the errors that do not require human attention, which frees attention for the ones that do.
\nThat framing \u2014 attention allocation rather than bug finding \u2014 is the one that makes it worth the setup.
\nMachine-generated code has a characteristic error profile: plausible, syntactically valid, and wrong in specific recurring ways. Unchecked errors, missing awaits, resource leaks, off-by-one in boundary conditions.
\nThose are exactly the errors static analysis is good at. Running a strict analyzer over generated code is verification you get for free, and it is one of the few places where the verification bottleneck has an automated answer.
\nTurn it on.
", "image": "https://readme.news/cards/static-analysis-is-finally-worth-the-false-positives.png", "date_published": "2026-06-05T09:00:00Z", "date_modified": "2026-06-05T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "tooling", "testing", "engineering", "craft" ] }, { "id": "https://readme.news/time-zones-and-why-your-calendar-code-is-wrong/", "url": "https://readme.news/time-zones-and-why-your-calendar-code-is-wrong/", "title": "Time zones, and why your calendar code is wrong", "summary": "An instant is not a time. A time is not a date. The rules change by government decree with a few weeks' notice.", "content_html": "Date and time handling is the domain where confident code is most reliably wrong, because the domain is defined by politics rather than physics and it changes without warning.
\nMost bugs come from conflating these.
\n1. An instant. A specific moment in the history of the universe. Stored as a UTC timestamp or an epoch value. \"The order was placed at 2026-06-03T14:22:11Z.\"
\nThis is what you want for: logs, audit trails, created-at, anything that records something that happened.
\n2. A local date and time with a zone. \"The meeting is at 09:00 on 2026-09-15 in Europe/Berlin.\"
\nCritically, this is not convertible to an instant in advance without risk, because the offset for Europe/Berlin on that date depends on the rules in effect at that time, and the rules can change between now and then.
\nThis is what you want for: future scheduled events, business hours, recurring appointments.
\n3. A plain date. \"2026-06-03.\" No time, no zone. A birthday. A contract date. An invoice period.
\nStoring a birthday as an instant is a bug. It will shift across the date boundary for users in some zones and someone will be wished a happy birthday on the wrong day.
\nStore instants in UTC. Always. timestamptz in Postgres, never timestamp. Convert at the display boundary.
Store future events as local time plus a zone identifier, not as a UTC instant.
\nThis is the one people get wrong most often, and it is the most consequential.
\nIf a user schedules a meeting for 09:00 on 15 September in Berlin, and you convert that to UTC now and store the instant \u2014 and Germany changes its daylight saving rules in July, which is a thing legislatures do \u2014 your meeting is now at the wrong time.
\nStore (\"2026-09-15T09:00\", \"Europe/Berlin\"). Convert at display time, using current rules.
Use IANA zone identifiers, never offsets. America/New_York, not UTC-5 and not EST.
An offset is a fact about one instant. A zone is a set of rules over time. Abbreviations are worse: CST is Central Standard Time, China Standard Time, and Cuba Standard Time, and there is no way to tell which.
Never do arithmetic on local times. Adding 24 hours to a local time is not the same as adding one day. On a DST transition day, one is 23 hours and the other is 25.
\nDecide which you mean:
\nThese differ twice a year and the bug reports are seasonal.
\nKeep your timezone database updated. The IANA database is updated several times a year because governments change rules, sometimes with weeks of notice.
\nIf your application bundles a tzdata snapshot from two years ago, it is wrong for several countries right now. This includes container images, JVM installations, and some language runtimes' bundled copies.
\nDST transitions create times that do not exist and times that occur twice. At a spring-forward transition, 02:30 local does not exist. At fall-back, 01:30 happens twice.
\nYour date library has a policy for this. Know what it is. The usual options are throw, shift forward, or pick the earlier occurrence, and the right one is application-specific.
\nLeap seconds. Mostly abstracted away by clock smearing at the infrastructure layer, and worth knowing exists. A minute is not always 60 seconds.
\nCountries change zones. Not just DST rules \u2014 entire offsets. Several countries have shifted their standard time in recent decades.
\n\"Midnight\" is ambiguous. Is 2026-06-03T00:00 the start or the end of 3 June? Prefer half-open intervals: [start, end). A day is from midnight inclusive to the next midnight exclusive. This eliminates an entire category of off-by-one.
The user's zone is not their locale is not their country. A user in Berlin may want dates in US format. Store these as separate preferences.
\nRecurring events are their own discipline. \"Every Tuesday at 09:00\" across a DST boundary means the UTC instant changes. Store the rule, not the instances, and expand at read time.
\nUse a real date library. Not string manipulation, not manual arithmetic. The edge cases are numerous, well known, and already handled by people who spent years on them.
\nIn JavaScript, the Temporal API finally provides the right distinctions natively \u2014 Instant, ZonedDateTime, PlainDate map exactly onto the three things above, which is why it took so long to design.
Test at boundaries. Your test suite should include a DST transition, a leap day, a year boundary, and a zone with a non-hour offset \u2014 India is UTC+5:30, Nepal is UTC+5:45, Chatham Islands is UTC+12:45. Code that assumes whole-hour offsets is common and wrong.
\nNever store or transmit local time without a zone. A timestamp without a zone is an ambiguous string and someone downstream will guess, and they will guess UTC, and they will be wrong by up to fourteen hours.
", "image": "https://readme.news/cards/time-zones-and-why-your-calendar-code-is-wrong.png", "date_published": "2026-06-03T09:00:00Z", "date_modified": "2026-06-03T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "engineering", "craft", "type-systems" ] }, { "id": "https://readme.news/rewrites-when-they-actually-work/", "url": "https://readme.news/rewrites-when-they-actually-work/", "title": "Rewrites: when they actually work", "summary": "The received wisdom is never rewrite. The received wisdom is mostly right and has three real exceptions.", "content_html": "The canonical advice is that rewriting from scratch is the single worst strategic mistake a software company can make. That advice is twenty-five years old and it is mostly still correct.
\nIt has exceptions, and knowing which situation you are in matters more than knowing the rule.
\nThe old system's behavior is undocumented and load-bearing. Every strange conditional in a decade-old codebase is a bug report somebody filed. You will not find them by reading the code, because the code does not say why. You will find them by shipping the rewrite and having those bugs re-reported.
\nThe rewrite has no users, so it gets no feedback. The old system is being exercised by real traffic continuously. The new one is exercised by your test suite, which encodes what you think it should do.
\nFeature parity is a moving target. The old system keeps getting features, because the business does not stop. You are chasing a target that recedes, and the chase consumes the time you budgeted for the rewrite.
\nNobody can justify continuing past month six. The rewrite has produced nothing users can see. The pressure to redirect the team to visible work is enormous and usually wins, leaving you with two systems.
\nThe knowledge is in the people, not the code. And the people who knew have frequently left, which is often why the rewrite was proposed.
\n1. The platform is dead.
\nThe framework is unmaintained, the language runtime is past end-of-life and getting CVEs, the vendor discontinued the product, the hardware is unobtainable.
\nThis is not a preference \u2014 you are being forced. Rewrite, and be grateful you found out before the security incident.
\n2. The domain model is fundamentally wrong.
\nNot \"the code is messy.\" The core abstractions do not correspond to reality, and every feature requires working around them.
\nThe test: can you name a specific business capability that is impossible, not just awkward, in the current model? \"We cannot support customers with more than one billing entity, and the fix requires changing what a customer is.\"
\nIf you cannot name one, you have messy code, not a wrong model, and messy code is fixed by refactoring.
\n3. The system is small enough that the rewrite is short.
\nIf the whole thing is three weeks of work, the calculus changes entirely. The risk of a three-week rewrite is bounded. Just do it.
\nThe received wisdom is about large systems, and people apply it to small ones where it does not hold.
\nStrangler fig, never big bang.
\nPut a routing layer in front of the old system. Move one endpoint at a time to the new implementation. Route traffic gradually. When everything has moved, delete the old system.
\n \u250c\u2500\u2192 new service (endpoints A, B)\nclient \u2192 router\n \u2514\u2500\u2192 legacy monolith (everything else)Properties this gives you:
\nRun both and compare. For a while, send traffic to both implementations, return the old one's response, and log the differences. This is the single most effective technique for discovering undocumented behavior, and it finds things no amount of code reading would have.
\nold = legacy.compute(req)\ntry:\n new = rewritten.compute(req)\n if new != old:\n log.warn(\"divergence\", request=req, old=old, new=new)\nexcept Exception as e:\n log.error(\"new path failed\", error=e, request=req)\nreturn old # old is still authoritativeRun that for weeks. The divergence log is your actual specification.
\nFreeze the old system's features. If the business keeps adding to it, you will never catch up. This requires an explicit organizational decision and it is usually the hardest part.
\nSet a deletion date and mean it. The worst outcome is two systems forever. If the migration stalls at 70%, you have doubled your maintenance surface permanently.
\nBefore any rewrite: what specifically will be better, and how will you know?
\nIf the answer is \"the code will be cleaner,\" that is not a business outcome and the project will lose its funding to something that is.
\nIf the answer is \"features in this area will take one week instead of three, and here are the last five that took three,\" you have a case.
\nThe rewrites that succeed are the ones where somebody could state the payback in a sentence. The ones that fail are the ones motivated by taste, and taste is not wrong \u2014 it is just not fundable.
", "image": "https://readme.news/cards/rewrites-when-they-actually-work.png", "date_published": "2026-06-01T09:00:00Z", "date_modified": "2026-06-01T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "architecture", "engineering", "craft", "opinion" ] }, { "id": "https://readme.news/what-a-great-bug-report-contains/", "url": "https://readme.news/what-a-great-bug-report-contains/", "title": "What a great bug report contains", "summary": "Six fields. Most reports have two. The difference is measured in days of engineering time.", "content_html": "A bug report's job is to get someone to the reproduction as fast as possible. Everything else is decoration.
\nMost reports fail at this, and the cost is enormous \u2014 a round trip asking for information takes hours or days, during which the bug is not being fixed and the reporter is not being helped.
\n1. What you did. The exact steps. Not \"I tried to log in\" \u2014 the sequence, including things that seem irrelevant. The irrelevant thing is frequently the cause.
\n2. What you expected. This seems redundant and is not. A meaningful fraction of bug reports are misunderstandings, and stating the expectation surfaces that immediately. It also catches the case where the software is working as designed and the design is wrong, which is a different and often more important bug.
\n3. What actually happened. Specifically. The exact error text, copied, not paraphrased and not described. Screenshots for visual issues. The full stack trace, not the last line.
\n4. Environment. Version of the software, operating system, browser, region, account type, relevant feature flags. Half of all \"cannot reproduce\" outcomes are environment differences.
\n5. Frequency. Every time? Once? Intermittently? This changes the debugging approach completely \u2014 a deterministic bug is a logic error, an intermittent one is usually concurrency, caching, or state.
\n6. An identifier. Request ID, trace ID, session ID, timestamp with timezone, account ID. This is what lets an engineer find the actual event in the logs, and it converts \"I have to reproduce this\" into \"I can look at exactly what happened.\"
\nThat last one is the highest-leverage field and the one most often absent, because most products do not surface an identifier to the user.
\nIf you build software: put a request ID on your error pages. It costs nothing and it is the difference between a bug report you can act on and one you cannot.
\nThe title is read by dozens of people and determines whether the right person opens it.
\nBad: \"Login broken\" Better: \"Login fails with 500 for SSO users after password reset\"
\nThe pattern: what fails, for whom, under what condition.
\nYour theory about the cause, unless you have evidence. A confident wrong theory sends the investigation in the wrong direction, and reports with a theory get debugged less carefully because the reader anchors on it.
\nIf you do have a theory, put it at the bottom, labeled as speculation.
\nEmotional content. \"This is completely broken and unacceptable\" adds nothing and makes the reader defensive. The severity is communicated by the impact description, not by adjectives.
\nMultiple bugs in one report. One issue per report. Multi-bug reports get partially fixed and closed, and the remaining bugs are lost.
\nIf you can produce one, it is worth ten times the rest of the report combined.
\nThe technique: start with the failing case and remove things until it stops failing. The last thing you removed is involved.
\nThis is work, and it is work that would otherwise be done by someone with less context than you. A minimal reproduction frequently reveals the cause to the reporter before they finish making it.
\nPut this in your issue template. Teams that do see a measurable improvement in report quality, because most people will fill in the fields you give them and will not invent them:
\n**What I did**\n1.\n2.\n3.\n\n**Expected**\n\n**Actual**\n(exact error text, full stack trace, screenshot)\n\n**Environment**\n- Version:\n- OS / Browser:\n- Account / Region:\n\n**Frequency**\nAlways / Sometimes / Once\n\n**Identifiers**\nRequest ID:\nTimestamp (with timezone):The other half of this.
\nReproduce before you theorize. The single most common debugging failure is building a mental model from the description and investigating that model instead of the actual behavior.
\nAsk for the missing field specifically. \"Can you send the request ID from the error page?\" gets an answer. \"Can you provide more details?\" does not.
\nClose the loop. Tell the reporter what it was. This costs a sentence and it is the reason people file good reports next time \u2014 a report that vanishes into a backlog teaches people not to bother.
\nA bug report is a handoff of context between two people, one of whom has seen the failure and one of whom has to fix it.
\nEverything in a good report is in service of moving context across that gap. Everything that does not move context across the gap is noise.
", "image": "https://readme.news/cards/what-a-great-bug-report-contains.png", "date_published": "2026-05-30T09:00:00Z", "date_modified": "2026-05-30T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "craft", "communication", "engineering" ] }, { "id": "https://readme.news/edge-computing-honestly/", "url": "https://readme.news/edge-computing-honestly/", "title": "Edge computing, honestly", "summary": "Running code close to users is a real win for a narrow set of workloads and a complication for everything else.", "content_html": "Edge computing has been sold as a general architectural improvement: run your code in hundreds of locations near users, everything gets faster.
\nThe physics is real. The applicability is narrower than the marketing, and the reason is data.
\nLight in fiber travels roughly 200,000 km/s. New York to London and back is about 55 ms of pure propagation, before any processing. Add TLS handshakes, TCP setup, and real-world routing that is not a great circle, and a cross-Atlantic round trip is frequently 100 ms or more.
\nRunning code 20 ms from the user instead of 120 ms is a genuine improvement and it is not achievable any other way.
\nYour data is not at the edge. It is in a database, in one region, and if your edge function needs it, you have moved the compute closer to the user and left the round trip in place \u2014 plus added a hop.
\nuser \u2192 edge (5ms) \u2192 origin database (120ms) \u2192 edge \u2192 userThat is slower than the user talking to the origin directly, because you added a hop to a path that was always dominated by the database call.
\nThis is the single most common edge computing mistake and it is easy to make, because the architecture diagram looks right.
\nAnything that needs no origin data:
\nAnything where the data is genuinely replicated to the edge:
\nSeveral platforms now offer edge-replicated key-value and SQL storage. If your data is small, read-heavy, and tolerant of replication lag \u2014 configuration, feature flags, product catalogs, translations \u2014 this works well and the latency win is real.
\nThe constraints are real too: writes go to a primary, replication is eventual, and storage per location is limited.
\nAnything write-heavy. Writes need coordination. Coordination needs a primary. The primary is in one place.
\nAnything requiring strong consistency. By definition, this needs coordination, which needs round trips, which is what you were trying to avoid.
\nAnything with a large working set. You cannot replicate a terabyte to three hundred locations.
\nAnything computationally heavy. Edge runtimes have tight CPU and memory limits. They are designed for milliseconds of work per request.
\nAnything that needs a specific runtime. Most edge platforms run a constrained JavaScript or WebAssembly environment. Your Python dependency with a C extension is not going there.
\nLayered, with each layer doing what it is good at:
\nedge \u2192 auth check, rate limit, routing, cached content, header work\nregional \u2192 application logic, caching, session state\norigin \u2192 the database, the writes, the truthMost requests are answered at the edge from cache. Some go to a regional application tier. Few reach the origin.
\nThat is a CDN with programmability, which is what edge computing actually is, and framing it that way produces much better decisions than framing it as \"serverless everywhere.\"
\nBefore adopting any of this: where does your latency actually go?
\nBreak down a typical request:
\nIf origin processing is 400 ms and network is 40 ms, moving compute to the edge addresses 40 ms of a 440 ms problem. Fix the 400 first.
\nThis is the most common reason edge adoption disappoints: it was applied to a latency problem that was not a network problem.
\nEdge is a very good CDN with programmability, and that is genuinely valuable \u2014 it lets you do real work at the perimeter that used to require an origin request.
\nIt is not a general application platform, and the platforms selling it as one are selling the constraint as a feature.
\nUse it for the perimeter. Keep your data where it can be consistent.
", "image": "https://readme.news/cards/edge-computing-honestly.png", "date_published": "2026-05-29T09:00:00Z", "date_modified": "2026-05-29T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "infrastructure", "architecture", "opinion" ] }, { "id": "https://readme.news/model-evaluation-for-people-who-ship/", "url": "https://readme.news/model-evaluation-for-people-who-ship/", "title": "Model evaluation for people who ship", "summary": "Not research benchmarks. A practical harness you can build in a day that makes every future model decision an hour instead of a week.", "content_html": "Every few weeks a new model ships and someone asks whether you should switch. Without an evaluation harness, answering that takes a week of impressions and you will get it wrong. With one, it takes an hour.
\nThis is the highest-return day of engineering available to anyone building on models, and a surprising number of teams have not done it.
\nNot MMLU. Not a leaderboard. Not \"vibes after twenty prompts.\"
\nPublic benchmarks tell you which models are worth testing. They do not predict performance on your task, because your task is not in them and because contamination is universal.
\nFifty to two hundred examples from your actual production traffic, with expected outputs or a grading rubric, run automatically, producing a number.
\nevals/\n cases/\n 001-refund-request.json\n 002-ambiguous-address.json\n ...\n run.py\n results/\n 2026-05-27-model-a.jsonEach case:
\n{\n \"id\": \"042\",\n \"input\": { \"ticket\": \"my order never arrived and I want my money back\" },\n \"expect\": { \"category\": \"refund\", \"urgency\": \"high\", \"needs_human\": false },\n \"notes\": \"the word 'never' should not trigger the fraud path\"\n}That is it. The whole thing is a test suite where the assertions are fuzzier.
\nYour production failures. This is the best source by a wide margin. Every time the system gets something wrong, that becomes a case. Your eval set grows into a precise map of your problem's difficulty.
\nSet this up as a workflow: a thumbs-down in the product, or a support escalation, creates a candidate case that someone reviews and adds.
\nYour edge cases. The weird inputs. The empty ones. The ones in another language. The adversarial ones. The ones with an injection attempt.
\nA stratified sample of normal traffic. So you notice when a change breaks the common case while fixing an edge case.
\nCases with no correct answer. Where the right behavior is to refuse, escalate, or ask a clarifying question. Models are frequently bad at this and it is rarely tested.
\nThree approaches, and you will use all three.
\nExact or structural match. For classification, extraction, and structured output. Cheap, deterministic, unambiguous. Use it wherever you can.
\nProgrammatic checks. For generated code: does it compile, do the tests pass. For SQL: does it run, does it return the right shape. This is the strongest form of grading and it is available more often than people realize \u2014 if you can verify mechanically, do.
\nModel-as-judge. For open-ended output. A second model grades against a rubric.
\nUse it carefully:
\nAccuracy on your set, obviously.
\nCost per case. Tokens in and out, at current prices. Track this \u2014 a model that is 2% better and 4\u00d7 the cost is usually the wrong choice.
\nLatency, at percentiles. p50 and p95. Reasoning models have high variance and the average hides it.
\nFailure mode distribution. Not just how many wrong \u2014 how wrong. A model that fails by refusing is very different from one that fails by confidently fabricating, and the aggregate score treats them identically.
\npython evals/run.py --model model-a --model model-b --parallel 8Output a table. Commit the results. Diff them across runs.
\nRun it:
\nThat last one catches something most teams never notice: the model you deployed against is not the model serving your traffic today.
\nVersion your prompts and store the eval result with them.
\nA prompt is production configuration. It should be in version control, reviewed, and associated with a measured quality number. \"Someone edited the prompt and something got worse three weeks ago\" is a debugging session that should not be possible.
\nOnce this exists:
\nIt is a day of work. It is the single highest-leverage day available in this space and it has been for three years.
", "image": "https://readme.news/cards/model-evaluation-for-people-who-ship.png", "date_published": "2026-05-27T09:00:00Z", "date_modified": "2026-05-27T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "ai", "testing", "engineering", "craft" ] }, { "id": "https://readme.news/idempotency-is-the-only-distributed-systems-concept-you-need/", "url": "https://readme.news/idempotency-is-the-only-distributed-systems-concept-you-need/", "title": "Idempotency is the only distributed systems concept you need", "summary": "Not the only one. But if you only internalize one, make it this one, because it makes most of the others survivable.", "content_html": "The network will deliver your message twice. Or zero times. Or once, but you will not find out, so you will send it again.
\nThis is not an edge case. It is the normal operating condition of every distributed system, and idempotency is what makes it survivable.
\nYou send a request. The connection times out.
\nDid it succeed?
\nYou do not know. There are three possibilities:
\nYou cannot distinguish these from the client. This is not a limitation of your tooling; it is a theorem about asynchronous networks. There is no protocol that resolves it.
\nThe only resolution is to make retrying safe.
\nAn operation is idempotent if performing it multiple times has the same effect as performing it once.
\nNaturally idempotent:
\nPUT /user/123 {\"name\": \"Dom\"} \u2014 set to a value.DELETE /user/123 \u2014 the second one finds nothing to do.SET balance = 100 \u2014 absolute assignment.Not idempotent:
\nPOST /orders \u2014 creates a new one each time.UPDATE accounts SET balance = balance + 100 \u2014 relative change.For operations that are not naturally idempotent, the client supplies a key and the server remembers it.
\nPOST /payments\nIdempotency-Key: 8f14e45f-ea5c-4b0d-9c1a-2f7e3d4a5b6c\n\n{\"amount_cents\": 4200, \"currency\": \"usd\", \"source\": \"card_x\"}Server side:
\ndef create_payment(key, request):\n with transaction():\n existing = lookup(key)\n if existing:\n if existing.request_hash != hash(request):\n raise Conflict(\"key reused with different parameters\")\n return existing.response # replay, do not re-execute\n\n result = charge_the_card(request)\n store(key, hash(request), result)\n return resultFour details that matter and are usually missed:
\nStore the result, not just the key. A retry should return the original response, not a \"already processed\" error. The client's retry should look like a successful first attempt.
\nHash the request. If the same key arrives with different parameters, that is a client bug and you should say so rather than silently returning the wrong result.
\nSame transaction. The lookup, the work, and the store must be atomic. Otherwise two concurrent retries both find nothing and both execute.
\nExpire the keys. Days, not forever. Storage is not free and a key from a year ago is not going to be retried.
\nMessage consumers. At-least-once delivery guarantees duplicates. Same pattern: an idempotency key on the message, a record of processed keys, both in one transaction.
\nWebhooks you send. Include an event ID. Your receivers will need it, and if you do not provide one they will invent something worse.
\nWebhooks you receive. Assume duplicates. Every major provider retries and several will send the same event twice under normal operation.
\nDeployment and provisioning. \"Create this resource\" should succeed if it already exists in the right state. This is why declarative infrastructure tools work and imperative scripts do not.
\nMigrations and backfills. They get interrupted. They must be safe to re-run from the beginning.
\nPrefer absolute over relative. SET balance = 100 is idempotent. ADD 100 is not. When you have the choice, take the absolute form.
Where you cannot \u2014 and you often cannot, because concurrent updates need relative operations \u2014 use a version or a compare-and-set:
\nUPDATE accounts SET balance = balance + 100, version = version + 1\nWHERE id = $1 AND version = $2;Zero rows updated means someone else got there first, and you know it.
\nLet the client generate the key. The client knows whether this is a new request or a retry. The server cannot tell. A server-generated key defeats the entire purpose.
\nDocument it. If your API supports idempotency keys, say so prominently, say how long they are honored, and say what happens on key reuse with different parameters. Consumers who do not know about the mechanism will not use it, and then they will double-charge someone and it will be your incident too.
\nMost distributed systems failures are one of: a duplicate, a message lost, or an out-of-order arrival.
\nIdempotency makes duplicates safe. Retries make loss survivable \u2014 and retries require idempotency to be safe. Which leaves ordering, which is a narrower problem that fewer systems actually have.
\nSo: one property, correctly implemented, defuses the majority of what goes wrong. That is a very good ratio, and it is why this is the thing to get right before anything else.
", "image": "https://readme.news/cards/idempotency-is-the-only-distributed-systems-concept-you-need.png", "date_published": "2026-05-26T09:00:00Z", "date_modified": "2026-05-26T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "architecture", "engineering", "craft" ] }, { "id": "https://readme.news/platform-teams-that-dont-get-resented/", "url": "https://readme.news/platform-teams-that-dont-get-resented/", "title": "Platform teams that don't get resented", "summary": "Internal platforms fail for predictable reasons. The successful ones share four properties.", "content_html": "Most internal platform teams end up resented by the engineers they serve. The pattern is consistent enough that the causes are identifiable.
\nEvery step follows from the previous one. The root is step four.
\n1. An escape hatch that is not punished.
\nThe platform covers the common case. It cannot cover every case, and pretending otherwise is what breaks trust.
\nThere must be a supported path for \"I need something the platform does not do,\" and taking that path must not require an exception process, a meeting, or an apologetic Slack message.
\nThe best platforms make the escape hatch cheap and then compete on being better than it. The worst make it forbidden, which does not eliminate the need \u2014 it drives it underground.
\n2. Adoption is voluntary, at least at first.
\nA platform that teams choose is a platform that is good. A platform teams are required to use never gets the feedback that would make it good, because the feedback mechanism \u2014 people leaving \u2014 has been disabled.
\nIf you cannot get voluntary adoption, that is information. Mandating it does not fix the underlying problem; it hides it and converts a product problem into a political one.
\nMandate later, when it is genuinely better, and the mandate will be uncontroversial because everyone already uses it.
\n3. The abstraction leaks deliberately, not accidentally.
\nEvery abstraction leaks. The question is whether you planned for it.
\nA good platform lets you drop a level when you need to: use the paved path for the deployment, and reach the underlying resource directly when you need something specific. A bad one hides the underlying system entirely, so that when it fails you cannot debug it and neither can the platform team, because now there are two systems to understand.
\nConcretely: if your platform generates infrastructure configuration, let people see it. If it wraps a cloud API, let people access the underlying resource. If it runs their container, give them the logs from the actual runtime, not a filtered view.
\n4. The platform team is measured on adoption and satisfaction, not on compliance.
\nIf the platform team's metric is \"percentage of services on the platform,\" they will optimize for mandating it.
\nIf the metric is \"would you use this if you had a choice,\" they will optimize for making it good.
\nAsk that question quarterly, anonymously, and publish the answer.
\nSlow escape. A team needs something the platform does not support. The answer is \"file a request, we will look at it next quarter.\" Their deadline is Friday.
\nBreaking changes without migration paths. The platform is infrastructure. Break it and every team stops. Platform teams frequently hold themselves to a lower compatibility standard than they would accept from a vendor.
\nOpaque failures. The deploy failed. The error is a platform-internal message. The product engineer cannot debug it and must escalate, which means waiting.
\nBeing a gate rather than a service. A platform that must approve things is a bureaucracy. A platform that makes the right thing easy is infrastructure.
\nSolving the platform team's problems. Standardization is valuable to the platform team and is not automatically valuable to product teams. If the pitch for a migration is \"this makes our lives easier,\" expect a cool reception.
\nYou are building a product. Your users are engineers. They have alternatives.
\nThat framing produces the right behaviors automatically: user research before building, documentation that assumes nothing, onboarding that works, support that responds, and a roadmap driven by what users need rather than by architectural preference.
\nThe platform teams I have seen work best behave exactly like a startup selling to a skeptical market, and they say so out loud.
\nThe ones that fail behave like an internal standards body, and they are usually correct about the standards and wrong about how to get them adopted.
\nTime from \"a new engineer joins\" to \"their code is running in production.\"
\nThat single number captures most of what a platform is for, it is measurable, and it is the thing product teams actually care about. If it is going down, the platform is working, regardless of what the adoption dashboard says.
", "image": "https://readme.news/cards/platform-teams-that-dont-get-resented.png", "date_published": "2026-05-22T09:00:00Z", "date_modified": "2026-05-22T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "engineering-management", "platforms", "tooling", "opinion" ] }, { "id": "https://readme.news/open-source-funding-five-models-honestly-compared/", "url": "https://readme.news/open-source-funding-five-models-honestly-compared/", "title": "Open source funding: five models, honestly compared", "summary": "Donations, foundations, dual licensing, open core, and hosted service. Each works for a specific shape of project.", "content_html": "The open source sustainability problem is not that the models do not exist. It is that projects pick a model that does not fit their shape, and then conclude that funding open source is impossible.
\nFive models. Each works. Each works for a specific kind of project.
\nWorks for: developer-facing tools with a large individual user base and an identifiable maintainer.
\nDoes not work for: libraries deep in a dependency tree, infrastructure nobody knows they use, anything without a personality attached.
\nThe honest arithmetic: donation income correlates with visibility, not with importance. A well-marketed CLI tool with a charismatic maintainer will out-earn a critical cryptography library by a large multiple.
\nThe corporate sponsorship version works better than individual donations and requires the maintainer to do sales, which most maintainers are bad at and hate.
\nWorks for: infrastructure that multiple large companies depend on and that none of them wants a competitor to control.
\nDoes not work for: small projects. The overhead \u2014 governance, legal, trademark, process \u2014 is substantial, and a foundation with one project and no funded staff is just more paperwork.
\nThe real value of a foundation is not money. It is neutrality: it makes a project safe for competitors to invest in together, which unlocks contribution that would not otherwise happen.
\nWorks for: libraries embedded in other products, where the copyleft obligation is genuinely inconvenient for commercial users.
\nShip under a strong copyleft license, sell a commercial license to companies that cannot comply.
\nDoes not work for: anything permissively licensed already (no leverage), anything not embedded (the obligation does not bite), or anything with a permissive competitor of similar quality.
\nEffective when it fits, and it produces a genuine tension: the license that makes the business work is the one that limits adoption.
\nWorks for: products where enterprise features are genuinely separable from the core \u2014 SSO, audit logs, RBAC, compliance reporting, multi-tenancy.
\nDoes not work for: libraries. There is no enterprise tier of a date-parsing library.
\nThe failure mode is well documented: the line between core and commercial moves toward commercial over time, under revenue pressure, and the community that built your adoption watches features they use get moved behind the paywall.
\nIf you do this, write down the line publicly, early, and honor it. \"Anything that a single developer needs is open; anything that exists because you have a compliance department is commercial\" is a defensible line. Moving it later costs more trust than the revenue is worth.
\nWorks for: anything that is annoying to operate. Databases, search, queues, observability, CI.
\nGive away the software, sell the operation of it. This is the strongest model when it fits, because the value you sell \u2014 not having to run it \u2014 is real and continuous, and it does not require withholding anything.
\nThe risk: a hyperscaler offers a managed version of your software, at scale, without contributing back. This has happened repeatedly and it is why the source- available licenses exist.
\nThose licenses solve the problem and cost you the open source designation, which costs you contributors, ecosystem inclusion, and some corporate adoption. It is a real trade with real costs on both sides, and the projects that made it mostly survived, which is the empirical answer to whether it works.
\nNot the absence of a model. Three other things:
\nSolo maintainer burnout. One person, unpaid, receiving an unbounded stream of issues, feature requests, and entitled demands. Funding helps and does not fix it \u2014 the fix is more maintainers, which is a governance problem.
\nSuccess without support. A project that becomes critical infrastructure while its maintainer count stays at one. This is the most dangerous state and it is extremely common.
\nCorporate abandonment. A company open-sources a project, staffs it with employees, then reorganizes. The external community was never built because it was never needed. Now nobody knows the code.
\nIf your business depends on open source \u2014 and it does \u2014 the highest-leverage actions, in order:
\nNone of that requires a strategy document. It requires someone with budget deciding it matters, which is the actual bottleneck.
", "image": "https://readme.news/cards/open-source-funding-five-models-honestly-compared.png", "date_published": "2026-05-20T09:00:00Z", "date_modified": "2026-05-20T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "open-source", "industry", "policy", "opinion" ] }, { "id": "https://readme.news/compression-is-underrated/", "url": "https://readme.news/compression-is-underrated/", "title": "Compression is underrated", "summary": "The cheapest performance win available, ignored because it is not glamorous. Where it pays and which algorithm to pick.", "content_html": "Compression trades CPU for bytes. On modern hardware, where CPU is abundant and bandwidth is the constraint at nearly every layer, that trade is favorable far more often than people apply it.
\nHTTP responses. Everyone does this and most do it badly. Check yours:
\ncurl -sI -H 'Accept-Encoding: br, gzip' https://example.com/api/data | grep -i content-encodingIf that returns nothing, you are shipping uncompressed JSON, and JSON compresses extraordinarily well \u2014 commonly 80\u201390% for typical API responses, because it is mostly repeated key names.
\nBrotli beats gzip by roughly 15\u201320% on text at comparable CPU cost, and is supported everywhere. Use it for static assets at maximum level (precomputed, so the CPU cost is paid once) and at a moderate level for dynamic responses.
\nDatabase storage. Most modern databases support per-table or per-column compression. On a table of text or JSON, it frequently halves the storage \u2014 which also halves the I/O, which means more of the working set fits in memory, which is where the real win is.
\nThe CPU cost of decompression is almost always smaller than the I/O cost you avoided.
\nLogs and telemetry. Log shipping is often a meaningful fraction of internal network traffic and vendor cost. Compressed batching typically reduces it by an order of magnitude, and the batching itself reduces request overhead.
\nBackups and object storage. Storage is cheap and it is not free, and egress definitely is not. Compression at rest is a direct cost reduction with no downside for cold data.
\nContainer images. Zstandard-compressed layers pull faster than gzip, which matters for cold starts and for anything that scales by launching new instances.
\nInter-service traffic. gRPC and Protocol Buffers are already compact. If you are sending JSON between services \u2014 and most people are \u2014 compressing it is a large win that costs one configuration line.
\n| algorithm | use for |
|---|---|
| zstd | almost everything. Wide speed/ratio range, fast decompression. |
| brotli | HTTP text, especially static assets at max level. |
| gzip | compatibility fallback. Never the best choice, always supported. |
| lz4 | when speed dominates entirely. In-memory, hot paths, real-time. |
| xz / lzma | archives you compress once and rarely read. Slow, small. |
The default answer is zstd. It has a level parameter spanning from faster-than-lz4 to nearly-as-small-as-xz, decompression is fast at every level, and it has dictionary support.
\nThe most underused feature in compression, and the one with the biggest payoff for small messages.
\nCompression works by finding repetition. A 200-byte JSON message has almost no internal repetition, so compression barely helps \u2014 sometimes it makes it bigger.
\nBut across many messages, there is enormous repetition: the same field names, the same enum values, the same URL prefixes.
\nA shared dictionary trained on representative samples gives the compressor that repetition up front:
\nzstd --train samples/*.json -o dict.zstThen compress each message against the dictionary. Small-message ratios that were 1.1\u00d7 become 3\u00d7 or better. For any system moving many small similar messages \u2014 event streams, queue payloads, cache values \u2014 this is a large and nearly free win.
\nAlready-compressed data. Images, video, audio, archives. You will spend CPU to make it slightly larger.
\nVery small payloads without a dictionary. Under a few hundred bytes, compression overhead can exceed the savings.
\nWhen you are CPU-bound and not bandwidth-bound. Measure. This is rarer than people assume but it does happen.
\nEncrypted data, compressed after encryption. Pointless \u2014 ciphertext is incompressible. And compressing before encryption can leak information about the plaintext through the ciphertext length, which is the class of attack that includes CRIME and BREACH. If you compress and encrypt, know why it is safe in your context.
\nThat is an afternoon and it routinely produces a larger improvement than a month of application-level optimization, at a fraction of the risk.
\nThe reason it does not happen is that nobody gets promoted for enabling Brotli.
", "image": "https://readme.news/cards/compression-is-underrated.png", "date_published": "2026-05-18T09:00:00Z", "date_modified": "2026-05-18T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "performance", "engineering", "infrastructure", "craft" ] }, { "id": "https://readme.news/the-queue-is-the-architecture/", "url": "https://readme.news/the-queue-is-the-architecture/", "title": "The queue is the architecture", "summary": "Most scaling problems are solved by making something asynchronous. Most reliability problems are caused by doing it badly.", "content_html": "The single most effective architectural move available to most systems is: take the slow thing out of the request path and put it in a queue.
\nIt is also the move that introduces the most subtle failure modes, and the gap between \"we added a queue\" and \"we added a queue correctly\" is large.
\nLatency. The user gets a response when the work is accepted, not when it is done. A checkout that returns in 80 ms and sends the confirmation email asynchronously is a much better product than one that returns in 900 ms.
\nAbsorbing spikes. A queue is a buffer. Traffic that would overwhelm a synchronous system accumulates and drains. This is the difference between a slow period and an outage.
\nIsolation. If the email provider is down, checkout still works. The messages accumulate and send later.
\nRetry for free. A failed message goes back on the queue. A failed synchronous call is a user-visible error.
\nEventual consistency, everywhere. The user completed checkout and the confirmation has not arrived. The record exists and the search index does not have it. Every asynchronous boundary introduces a window where the system is inconsistent, and your UI has to be honest about it.
\nDebugging across the boundary. A synchronous stack trace tells you the whole story. An asynchronous failure requires correlating a producer, a broker, and a consumer, possibly hours apart.
\nOrdering. Most queues do not guarantee it, or guarantee it only within a partition. If your consumer must process events in order, that is a design constraint that reaches back into how you partition.
\nDuplicate delivery. Almost all queues are at-least-once. Your consumer will receive the same message twice. If that is not safe, you have a bug that appears under load, weeks after launch.
\n1. Consumers must be idempotent. Non-negotiable.
\nAt-least-once delivery means duplicates. The consumer must produce the same result whether it processes a message once or five times.
\nThe usual implementation: a natural idempotency key, and a record of processed keys.
\ndef handle(msg):\n key = msg.idempotency_key\n with tx():\n if already_processed(key):\n return\n do_the_work(msg)\n mark_processed(key)The mark_processed must be in the same transaction as the work, or you have moved the race rather than eliminated it.
2. Every queue needs a dead letter queue, and someone must watch it.
\nA message that fails repeatedly must go somewhere. A DLQ nobody monitors is a place where data goes to be silently lost, which is worse than an error, because errors are visible.
\nAlert on DLQ depth. Not on it being non-zero \u2014 on it growing.
\n3. Retry with backoff and jitter, and cap the attempts.
\nImmediate retry on a failing downstream is a denial of service you are performing against yourself. Exponential backoff with jitter, a maximum attempt count, then the DLQ.
\n4. Monitor queue depth and age, not just throughput.
\nThroughput looks healthy right up until it does not. The metrics that tell you something is wrong:
\n5. Decide what happens when the queue is full.
\nIt will be. Reject the producer, drop messages, or block? Each is right in different cases and the default is usually wrong for you. An unbounded queue is not a solution; it is a memory leak with extra steps.
\n6. Keep the payload small and the reference stable.
\nPut an ID in the message, not the whole object. The consumer fetches current state. This avoids stale data in the message and keeps the broker fast.
\nThe exception: if you need the state as it was when the event occurred, put it in the message deliberately, and say so.
\n7. Version your message schema from day one.
\nProducers and consumers deploy independently. Old consumers will see new messages. Include a version field. Make additive changes only, or handle both shapes.
\nQueues do not reduce load. They defer it.
\nIf your consumer processes 100 messages per second and you produce 150, you do not have a working system with a buffer. You have a system that is failing slowly, and the queue depth graph is a countdown.
\nA queue absorbs bursts. It does not fix a sustained capacity deficit, and the failure mode when you use it that way is a queue that grows for six hours and then an incident where you are simultaneously behind and unable to catch up.
\nAlert on the age, watch the trend, and size the consumers for the sustained rate.
", "image": "https://readme.news/cards/the-queue-is-the-architecture.png", "date_published": "2026-05-15T09:00:00Z", "date_modified": "2026-05-15T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "architecture", "infrastructure", "reliability", "engineering" ] }, { "id": "https://readme.news/run-the-incident-before-the-incident/", "url": "https://readme.news/run-the-incident-before-the-incident/", "title": "Run the incident before the incident", "summary": "Game days, failure injection, and the specific reason your untested runbook is wrong.", "content_html": "Every runbook that has never been executed is wrong. Not might be \u2014 is. The commands have changed, the dashboard moved, the person who wrote it left, and the system it describes has been modified fourteen times.
\nThe only way to find out is to run it, and the only good time to run it is when nothing is actually broken.
\nA scheduled exercise where you deliberately break something in a controlled way and have the on-call rotation respond as if it were real.
\nTwo hours. A stated scenario. The people who would actually respond. Real tools, real runbooks, real dashboards. Someone taking notes on everything that did not work.
\nThat is the whole practice. It is not chaos engineering in the automated-random sense, though that is a good adjacent practice. It is a rehearsal.
\nI have never run one of these that did not find at least four of the following:
\nThe runbook references something that does not exist. A dashboard that was renamed, a script in a repository that was archived, an alias nobody has.
\nNobody has the access. The runbook says to restart the service. The on-call engineer does not have permission and does not know who does. This is the single most common finding.
\nThe dashboard does not show the thing. You built the alert. You never built the view that tells you what to do about it.
\nThe escalation path is a person, not a rotation. \"Ask Sarah.\" Sarah is on vacation. Sarah left last year.
\nNobody knows the customer impact. The service is degraded. Which customers? Which features? Nobody can answer, so nobody can decide how urgent it is.
\nRecovery has an undocumented step. The service restarts and does not work because a cache must be cleared first, which is knowledge that lives in one person's head.
\nThe communication path is unclear. Who tells the customers? Who updates the status page? Who has the credentials for the status page?
\nEvery one of those is cheap to fix on a Tuesday afternoon and extremely expensive to discover at 3 a.m.
\nStart with the boring ones. Exotic failure modes are fun and the common ones are what actually happens.
\nThe database primary fails over. Does the application reconnect? How long? Does anything need a manual restart?
\nA dependency returns 500s. Not down \u2014 erroring. Does your circuit breaker work? Does your retry policy make it worse?
\nA dependency gets slow. Harder than down and much more common. Does your timeout fire? Do connections exhaust? Does the slowness propagate to callers?
\nDisk fills up. On the database host, on the log host, on the application host. This is the single most common self-inflicted outage.
\nA bad deploy. Deploy something broken to staging and time the rollback. Not the theoretical rollback \u2014 the actual one, executed by the actual on-call person.
\nCertificate expiration. Set one to expire in staging. Watch what happens. Most teams discover their monitoring does not cover this.
\nThe person who knows is unavailable. Run a game day where the subject matter expert is explicitly not allowed to help. This finds the knowledge concentration problems that nothing else does.
\nAnnounce it. Do not surprise people. Surprise exercises generate resentment and teach people to distrust the process. Everyone should know it is a drill.
\nStaging first, production eventually. Staging finds most of the runbook problems. Production finds the ones that only exist because staging is not production, which are real and are the ones that matter most.
\nHave a stop condition. A named person who can call it off, and a defined way to revert whatever you broke.
\nWrite down every friction point, including small ones. \"It took four minutes to find the right dashboard\" is a real finding.
\nFix things within a week. A game day that produces a list nobody acts on is theater, and the second one will have lower attendance.
\nDo it quarterly. Systems change. A runbook validated a year ago is a runbook that has not been validated.
\nThe purpose is to find gaps in the system, not gaps in people.
\nIf someone cannot resolve the scenario, that is a finding about documentation, tooling, or access \u2014 not about them. Say this explicitly before you start, and mean it, or people will optimize for looking competent rather than for surfacing problems.
\nThe best outcome of a game day is a long list of things that went wrong, discovered by people who were not stressed, on a schedule, with time to fix them.
", "image": "https://readme.news/cards/run-the-incident-before-the-incident.png", "date_published": "2026-05-13T09:00:00Z", "date_modified": "2026-05-13T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "reliability", "engineering", "testing", "craft" ] }, { "id": "https://readme.news/python-315-and-free-threadings-second-act/", "url": "https://readme.news/python-315-and-free-threadings-second-act/", "title": "Python 3.15 and free-threading's second act", "summary": "The GIL-free build is officially supported and the ecosystem work is the actual story. A progress report.", "content_html": "Python 3.15 is in beta. The headline features are incremental. The story worth tracking is the free-threading migration, which is now in the phase where it succeeds or stalls based on ecosystem work rather than on CPython.
\nThe GIL-free build has been officially supported since 3.14. The question was never whether CPython could do it \u2014 it demonstrably can \u2014 but whether the ecosystem would follow.
\nWhat has gone well:
\nThe major numerical and data libraries did the work. NumPy, and the array and dataframe ecosystem around it, ship free-threaded wheels. That was the critical path, because those libraries are the reason a large fraction of Python's CPU-bound workload exists.
\nSingle-threaded performance in the free-threaded build has improved substantially since 3.13. The gap against the default build has narrowed to something most applications would not notice.
\nBuild infrastructure caught up. Producing free-threaded wheels is now a configuration change rather than a project.
\nWhat has gone slowly:
\nThe long tail. Thousands of packages with C extensions have not been audited, and most of them will not be until someone hits a problem.
\nThe subtler issue: pure-Python code that was accidentally thread-safe. The GIL made many operations effectively atomic. Code written under that assumption \u2014 a dictionary mutated from multiple threads, a counter incremented without a lock \u2014 worked by accident. Under free-threading it does not.
\nNobody knows how much library code depends on this, because nobody ever had to think about it. It will be discovered one race condition at a time.
\nDo not switch production workloads casually. The performance win only exists for CPU-bound multithreaded work. If your workload is I/O-bound, asyncio was already handling it and free-threading gives you nothing.
Do test your libraries against it. The compatibility work has to happen and it happens when people run into problems and report them. If you maintain a package with a C extension, this is your work to do.
\nDo use it for the workloads it is for. Data processing, image and signal processing, simulation, anything numeric that currently uses multiprocessing and pays serialization and memory-duplication costs.
The migration from multiprocessing to threads for those workloads is frequently dramatic \u2014 not just faster, but simpler, because you delete the serialization layer.
Subinterpreters continue maturing. concurrent.interpreters gives you isolated interpreter instances in one process, with much lower overhead than processes and much stronger isolation than threads.
This is the underrated option. For a workload where you want parallelism and do not want shared mutable state \u2014 which is most workloads \u2014 subinterpreters give you the process model's safety at closer to the thread model's cost.
\nIt is the right default for a lot of the cases people currently reach for multiprocessing for, and it does not require any of the thread-safety auditing that free-threading does.
Error messages continue improving, which has been a multi-release trend and has made Python meaningfully friendlier for people learning it.
\nPython is executing a decade-long project to stop being single-threaded, and it is doing it without breaking the ecosystem, on an opt-in basis, with a fallback.
\nThat is the right way to do it and it is slower than anyone wants. The alternative \u2014 a hard break, a Python 4 \u2014 would have been faster and would have cost the community what Python 3 cost it, which nobody has the appetite for.
\nAsk again in three years. The trajectory is good and the destination is not in doubt; only the timeline is.
", "image": "https://readme.news/cards/python-315-and-free-threadings-second-act.png", "date_published": "2026-05-11T09:00:00Z", "date_modified": "2026-05-11T09:00:00Z", "authors": [ { "name": "Dom the Developer", "url": "https://domthedeveloper.com" } ], "tags": [ "python", "languages", "releases" ] } ] }