Enough time has passed to say what actually happened rather than what everyone predicted.

what changed#

The norm inverted. Before, crawling was permitted by default and robots.txt was a request. Now, for a large share of the web, crawling is denied by default and access is a negotiation.

That is a genuine structural change to how the web works and it happened through one company's configuration default rather than through any standards process, legislation, or public deliberation.

Licensing deals concentrated. Large AI companies negotiated bulk access with large publishers. That was always the likely outcome: the parties with lawyers and leverage made arrangements, and the arrangements are private.

Small publishers got very little. The pay-per-crawl mechanism works, technically. The revenue for a site with modest traffic is negligible — the arithmetic never supported anything else. The publishers who most needed a new economic model got the one that pays least.

Non-commercial crawling got harder. Academic researchers, archivists, and independent search projects have no licensing budget and no negotiating position. Carve-outs exist and are discretionary, which means the ability to study the web is now something you apply for.

That is the outcome I was most worried about and it is the one that materialized most clearly.

what did not change#

Training data supply. The frontier labs have enormous existing corpora, licensed sources, and synthetic generation. The marginal value of newly crawled web text was already declining. Restricting it did not create the leverage publishers hoped for.

Traffic. Referral traffic from search to publishers continued its decline, driven by AI answers in search results, which is a completely separate mechanism from training crawlers and which blocking crawlers does nothing about.

This is the part that was most misunderstood at the time. The traffic problem and the training problem have different causes and blocking crawlers only addresses one of them — the one with less economic impact.

the thing to actually take from it#

Infrastructure defaults are policy. A configuration change at a chokepoint reshaped access to a large fraction of the web, with no process and no appeal.

That is not a criticism of the specific decision, which was popular and defensible. It is an observation about where power actually sits, and it generalizes: the entities that can change the web's behavior are the ones with concentration at a layer everyone depends on, and there are about five of them.

For your own site, the decision remains yours and it is worth making deliberately rather than accepting a default:

• Documentation sites frequently want to be in the training data. Being the thing the model knows about is worth more than the pageview you did not get.
• Original reporting and analysis has a stronger case for restriction.
• Anything you want found should still permit search crawlers, which are a different category and are frequently blocked by accident when people configure this.

Check what you are actually blocking. A meaningful number of sites blocked their own search indexing in the first months of this and did not notice for weeks.

the unresolved thing#

The web's economic model — publish freely, get traffic, monetize traffic — is breaking, and nothing has replaced it.

Crawler tolls are not the replacement; the arithmetic does not work at the scale of the actual web, where most content is made by people with no ability to negotiate anything.

Licensing deals are not the replacement either; they work for a few hundred large publishers and for nobody else.

I do not know what the replacement is. I am increasingly convinced that nobody does, and that the interval between the old model failing and a new one existing is going to be long and is going to be bad for the open web.

That is a genuinely pessimistic conclusion and I have not found a way around it in a year of thinking about it.

Having run this for a while at meaningful volume, here is what actually happens.

what works#

Mechanical, well-specified, verifiable work. This is not a hedge, it is the finding. The tasks where fleets genuinely deliver:

• Dependency upgrades across many services.
• Migrating a deprecated API call across a large codebase.
• Adding tests to modules with poor coverage.
• Converting between formats or frameworks with a mechanical mapping.
• Fixing a class of lint or type error across a repository.

What these share: a clear acceptance criterion the agent can check itself, a bounded scope, and no design decisions.

For this category the leverage is real and large. Work that would have been a week of tedium becomes an afternoon of review.

Investigation in parallel. Spawning several agents to independently investigate a bug from different angles — read the logs, bisect the history, read the related code, reproduce it — and reading all four reports is genuinely faster than doing them serially. The agents are cheap; your attention is not; parallelizing the cheap thing is correct.

what does not work#

Anything requiring shared context. Five agents working on related parts of a system produce five changes that each make sense and collectively do not. They duplicate helper functions. They pick different names for the same concept. They each add a slightly different retry wrapper.

You end up with a merge problem that is worse than the original work, because the conflicts are semantic rather than textual — the diffs apply cleanly and the result is incoherent.

Anything with genuine design decisions. An agent given an ambiguous task will resolve the ambiguity confidently and silently, and it will pick a reasonable option that may not be the one you wanted. Five agents will each pick differently.

Anything where the acceptance criterion is subjective. "Improve the error messages" produces five different notions of improved.

the actual bottleneck#

Review, exactly as predicted, and worse than predicted.

Five pull requests per hour is not a throughput improvement if you can meaningfully review two. What you get is a queue, and queue pressure degrades review quality in a way that is invisible in the metrics and visible in the defect rate a quarter later.

The honest arithmetic: your throughput is min(generation rate, review rate), and review rate did not change.

what actually helps#

Make the agent produce a reviewable artifact, not just a diff. A summary of what it did, what it decided, and what it was unsure about. Reviewing "I chose to use the existing retry helper rather than adding a new one, and I left the timeout at 30s because the surrounding code does" is dramatically faster than inferring that from a diff.

Batch related work into one agent, not many. If five tasks touch the same module, one agent doing all five produces a coherent change. Five agents produce five incoherent ones. Parallelism across independent work only.

Invest heavily in verification. A strong test suite is what makes review cheaper, because you are reviewing design rather than correctness. Teams with weak tests get no benefit from agent fleets — they just get more code to verify by reading.

Cap concurrency at your review capacity. Running more agents than you can review does not help. It produces a backlog that goes stale and gets abandoned, which is worse than not starting.

Reject on size. A machine can generate a 2,000-line diff effortlessly. Say no. Ask it to split.

the number#

For our work, the useful concurrency turned out to be two to three agents on independent tasks, with one person reviewing. Beyond that, quality degraded and the extra output was not landing.

That is a real multiplier and it is much less than the demos suggest, and the constraint is entirely on the human side.

the thing I would tell someone starting#

Do not start with a fleet. Start with one agent, on the mechanical work you have been putting off, and measure whether the output actually lands.

If your review process cannot absorb one agent's output, adding four more is solving the wrong problem.

It is still worth understanding what it actually changed, because a few of the consequences affect how you build.

the problem it solved#

HTTP/2 introduced multiplexing: many logical streams over one TCP connection. That fixed head-of-line blocking at the HTTP layer.

It did not fix it at the TCP layer. TCP delivers bytes in order. If one packet is lost, everything behind it waits, including data for streams that were completely unaffected. On a lossy connection — mobile, congested wifi — HTTP/2 could be worse than HTTP/1.1 with six connections, because one loss stalled everything instead of one sixth of everything.

QUIC moves the transport to UDP and implements reliability, ordering, and congestion control per stream. A lost packet stalls only the stream it belonged to.

what else came with it#

Encryption is mandatory and integrated. TLS 1.3 is part of the protocol rather than a layer on top. The handshake is one round trip, or zero for a resumed connection.

Connection migration. A QUIC connection is identified by a connection ID, not by the four-tuple of IP addresses and ports. Change networks — wifi to cellular — and the connection survives. Your download does not restart.

This is the feature users notice without knowing why. Walking out of a building while a video plays used to stall it.

Better loss recovery. QUIC distinguishes between packet loss and reordering more accurately than TCP, and its acknowledgment format carries more information. Recovery is faster.

Evolvability. TCP is implemented in kernels and middleboxes and cannot change, because the internet is full of devices that will drop anything unfamiliar. QUIC is in userspace and encrypted, so its internals are invisible to middleboxes and can actually be updated.

That last point is arguably the most important long-term consequence. Transport protocol ossification was a genuine crisis and QUIC is the escape hatch.

the practical consequences for you#

Domain sharding is now actively harmful. Splitting assets across static1.example.com and static2.example.com was a workaround for HTTP/1.1's connection limit. Under HTTP/2 it was pointless. Under HTTP/3 it is worse than pointless, because each domain requires a separate connection with a separate handshake and separate congestion state.

One origin. If you still have sharding from a 2014 optimization guide, remove it.

Concatenating and spriting are counterproductive. Same reasoning. Many small files multiplex fine and cache better individually.

Priority matters and is under-configured. HTTP/3 has an extensible priority scheme. Most servers use defaults. If you have a page where certain resources are critical, priority hints (fetchpriority) are worth setting and are widely supported.

UDP blocking is real but small. Some corporate networks block UDP/443. Clients fall back to HTTP/2 automatically, so this is a performance issue rather than a correctness one. Do not build anything that requires HTTP/3.

Your observability may not see it. A lot of network monitoring tooling was built for TCP. Check whether your tools actually understand QUIC or are silently reporting nothing.

the parts that were harder than expected#

CPU cost. QUIC's userspace implementation and per-packet encryption use more CPU than kernel TCP. This has improved substantially with offload support and better implementations, and it is a real cost at high volume.

Middlebox hostility. Some networks throttle or block UDP because it looks like something they should throttle. This is improving as QUIC becomes normal traffic.

Debugging is harder. You cannot read a QUIC connection with tcpdump the way you could read HTTP/1.1. qlog and browser devtools help. The tooling is younger.

the assessment#

For a user on a good connection, HTTP/3 is roughly a wash. For a user on a bad connection — mobile, congested, high latency, lossy — it is a substantial improvement, and those users are a large fraction of the world.

That is exactly the right kind of improvement: invisible to the people who were already fine, meaningful to the people who were not.

Enable it, remove your HTTP/1.1-era workarounds, and go back to not thinking about the transport layer.

what they are#

A virtual thread is a thread managed by the JVM rather than the operating system. Creating one costs on the order of a few hundred bytes rather than a megabyte of stack. Blocking one parks it and frees the underlying carrier thread rather than blocking an OS thread.

The consequence: you can have millions of them, and blocking is no longer expensive.

try (var executor = Executors.newVirtualThreadPerTaskExecutor()) {
    for (var request : requests) {
        executor.submit(() -> handle(request));   // one thread per request, fine
    }
}

why this mattered more than it looked#

The Java ecosystem spent a decade building around the assumption that threads are expensive. That assumption produced:

• Thread pools everywhere, with sizing that nobody understood.
• Reactive programming frameworks, with their entirely separate ecosystem of types, operators, and debugging misery.
• CompletableFuture chains that are unreadable after three composed steps.
• Stack traces that told you nothing because the actual work happened on a different thread than the code you wrote.

All of that complexity existed to avoid blocking. Virtual threads make blocking cheap, which removes the reason for all of it.

The single biggest practical win is debugging. A virtual thread's stack trace is the actual call stack of the actual logical operation. Compare that to a reactive pipeline, where the stack trace is the scheduler and the real causal chain is reconstructed by squinting at operator names.

Teams that migrated report this as the thing they did not expect and would not give up.

the pitfalls that are real#

Pinning. A virtual thread inside a synchronized block cannot unmount from its carrier thread — it pins it. If that code then blocks, you have consumed an OS thread for the duration, and with a small carrier pool you can deadlock.

The fix is ReentrantLock instead of synchronized. Later JDK releases reduced the pinning cases substantially; library code you do not control can still do it.

Detect it:

-Djdk.tracePinnedThreads=full

Run that in a load test before you go to production. Every migration finds something.

ThreadLocal at scale. A ThreadLocal with a million threads is a million copies. Libraries that cache expensive objects per-thread — some serializers, some date formatters, some connection helpers — become a memory problem.

Scoped values are the intended replacement and are the right answer for request-scoped context.

Unbounded concurrency. Thread pools were an accidental rate limiter. Remove them and you can now issue ten thousand concurrent requests to a downstream service that handles two hundred. You have moved the failure from your service to theirs, which is worse.

You still need bounded concurrency. It just belongs at the resource — a semaphore around the downstream call — rather than as a global thread pool.

Connection pools. Your database connection pool is sized for a thread-pool world. It is now the bottleneck, and the right size is a different calculation. This surprises people.

the migration advice#

Do not rewrite anything. Virtual threads work with the code you have. Switch the executor, run your load tests, look for pinning, size your connection pools, done.

Do not migrate reactive code that works. A well-functioning reactive service should be left alone. The migration cost is real and the benefit is maintainability, which is a slow-compounding return.

Do migrate new services. Writing a new service with virtual threads and plain blocking code is dramatically simpler than the reactive equivalent, and simpler code is the whole point.

the broader lesson#

An enormous amount of accumulated architectural complexity existed to work around one runtime limitation. Removing the limitation made the complexity obsolete overnight — but only for code written after, because nobody rewrites working systems.

That is worth remembering the next time you build elaborate machinery around a platform constraint. Ask how likely the constraint is to persist, and whether the machinery will outlive it.

Usually it will, sitting in your codebase, load-bearing and pointless.

what it does#

Historically Postgres read from disk synchronously: a backend process issues a read, blocks, gets the page, continues. On a spinning disk that was fine — the disk was the bottleneck and you could not do better. On modern NVMe, where the device can service many requests in parallel and a single-threaded synchronous reader leaves most of the device idle, it wasted a great deal of available throughput.

Postgres 18 adds an I/O method abstraction:

io_method = sync          # the old behavior
io_method = worker        # dedicated I/O worker processes (default)
io_method = io_uring      # Linux io_uring, if built with support

Sequential scans, bitmap heap scans, and vacuum use it. Other paths do not yet; this is an incremental rollout across releases.

what people are seeing#

The consistent pattern from the reports I trust:

Large sequential scans on fast NVMe: substantial improvement. Analytical queries scanning a large table are the clearest win. This is exactly what you would predict — the workload was read-parallel and the code was not.

Vacuum: meaningfully faster. This matters more than it sounds. Vacuum being slow is the root of a large class of Postgres operational problems, and anything that makes it finish sooner reduces bloat pressure.

OLTP with a good cache hit rate: little change. If your working set is in shared buffers, you were not doing I/O, so making I/O faster does nothing. Most transactional workloads are in this category and should not expect much.

Cloud block storage: mixed. Network-attached storage has enough latency and enough of its own queueing that the gains are smaller and less predictable than on local NVMe. Test rather than assume.

the tuning notes#

Two settings matter and the defaults are conservative:

io_combine_limit = 128kB        # how much adjacent I/O to merge into one request
effective_io_concurrency = 16   # how many concurrent reads to issue

effective_io_concurrency has historically been set based on spindle count, which is meaningless on NVMe. For a modern SSD, values well above the old defaults are appropriate. Measure — the curve flattens and then degrades, and where it turns depends on your device.

If you are on Linux with a recent kernel and can build with io_uring support, it generally outperforms the worker method by a modest margin. The worker method is the safe default and it is genuinely good.

the other things in 18 worth knowing#

Skip scan for B-tree indexes. A multi-column index on (a, b) can now be used for a query filtering only on b, when a has low cardinality. This eliminates a category of redundant index that people have been maintaining for years.

uuidv7() built in. Time-ordered UUIDs, natively. If you use UUIDs as primary keys, v7 dramatically improves index locality compared to v4, which scatters inserts across the whole B-tree. This is a real performance issue that a lot of applications have and do not know about.

OAuth authentication support, which matters for anyone trying to eliminate static database passwords.

Faster major-version upgrades, with statistics preserved across pg_upgrade. Previously you finished an upgrade with no statistics and a database that planned terribly until you ran ANALYZE across everything. That was a genuine outage risk and it is fixed.

should you upgrade#

Yes, on the normal schedule — after a point release or two, with a tested rollback, having read the release notes for the incompatibilities.

The async I/O is the headline and for most transactional workloads the statistics preservation on upgrade and uuidv7() will matter more day to day.

As always with Postgres: the release is well-tested, the migration is usually boring, and the people who get burned are the ones who skipped four major versions and tried to do it in one jump.