The New Gold Rush in AI: Loop Engineering

In the 1780s James Watt bolted a pair of spinning brass balls to a steam engine so it could hold its own speed. When the engine sped up, the balls flew outward, pinched the steam valve, and slowed it back down. A machine correcting itself against its own output. In 1867 Maxwell wrote down the mathematics of when that little dance stays smooth and when it starts to hunt and shake itself apart.

That is loop engineering. It is 240 years old.

So it’s a little strange to watch the AI field talk about self-observing, self-revising systems as if we’d just invented the wheel. We didn’t. We inherited a very large, very well-documented library on exactly this problem — from control theory, cybernetics, biology, economics — and most of it is a warning.

Here’s the through-line, and I’ll say it plainly because everything else hangs off it: a feedback loop that judges its own output, with nothing outside it to check against, does not settle into a happy equilibrium. Its default behavior is to oscillate, to diverge, or to collapse onto some trivial answer that games its own scoring. Stability is the exception. It has to be built in on purpose, and we know a fair amount about how.

But there’s a second half to that sentence, and it’s the one people skip. Bolting some external anchor onto the loop isn’t the win. The win is bolting on the right one — an anchor that actually encodes what you’re trying to achieve. Get that choice wrong and the loop doesn’t break down where you can see it. It works beautifully, and it optimizes hard in a direction you never intended. A loop is only ever as good as the thing it’s pointed at.

Start with the tools, because they already exist. By the 1930s and 40s, Nyquist, Bode and Black had rigorous ways to look at a loop and say whether it would converge or tear itself apart. Ashby, in 1956, added a constraint that ought to be pinned above every loop-engineering desk: the Law of Requisite Variety. A controller must command at least as much corrective range as the thing it’s trying to control has room to misbehave. Put a thin, dumb evaluator in charge of a rich, capable generator and you have already lost — the controller simply cannot see enough of the ways the system can go wrong. A lot of “let the model grade itself” designs violate this on day one.

Does that sound abstract? It stops being abstract the moment you look at who has actually run these loops for real, at scale, with lives attached.

Aerospace has the deepest record. Fly-by-wire aircraft and the Apollo guidance computer ran real-time correction loops with certified stability margins — formally verified to hold together within a specified range of disturbance. Chemical plants have run feedback control since the 1940s and have eighty years of incident reports to prove what happens when a loop’s model drifts from reality; Texas City in 2005 is one of the grim entries. Power grids are the largest feedback systems ever built, and the 2003 blackout that took out the American northeast was a cascade of loops failing into each other. Algorithmic trading learned it fastest and hardest: the 2010 flash crash was feedback between machines going unstable in seconds, and the industry’s response — circuit breakers, forced pauses — is nothing but engineered damping bolted on after the fact.

Notice what every one of the stable systems has in common. Watt’s governor works because it does exactly one thing in a narrow, well-understood domain. Fly-by-wire works because formal verification forbids it from rewriting its own goals. The plant controller works because a human can override it. Bounded scope, a clear definition of success, and human oversight of the loop itself. Those aren’t nice-to-haves. On the historical evidence they are the whole ballgame.

Now point that lens at the three places AI wants to run loops today, and ask the honest question: experience, or virgin territory?

Business workflow has the most experience and the least of the kind we need. Deming’s Plan-Do-Check-Act cycle from the 1950s — the engine of the Toyota production system — is a genuine self-improving loop that cut real defects, and it stayed stable because it had damping and a human in it. But most corporate feedback loops are Goodhart machines. Goodhart’s Law, from 1975: the moment a measure becomes a target, it stops being a good measure. People optimize the number, not the thing the number was supposed to stand for. This is not a quirk. It is the single most reliable finding in the entire literature of feedback-driven improvement, documented in schools teaching to the test, in hospitals gaming quality scores, in every quarterly-earnings distortion you’ve ever read about. Reward hacking in AI is just Goodhart with a gradient descent attached.

Take the most mundane example there is. Point an optimization loop at a website and anchor it on a single number — traffic — and it will give you traffic, faithfully and relentlessly: clickbait titles, rage-bait, a photo gallery sliced into fourteen pages. What it will not necessarily give you is more business. The anchor was real, external, and measurable, and it still pointed the loop at the wrong hill. Anchor the same loop on a balanced plateau instead — returning visitors and time-to-purchase and actual revenue and refund rate — and the identical machinery pulls somewhere completely different. Nothing about the loop changed. Only what it was told to hold onto. That choice, more than the cleverness of the algorithm, decides whether you drift and which way.

Software engineering is closer to home and more encouraging — up to a point. Continuous integration and deployment are working feedback loops, and the DevOps data is clear that they improve things. But read the same data carefully and the lesson is the opposite of “loops self-stabilize.” They stabilize because someone engineered explicit rollback thresholds and circuit breakers into them. The equilibrium was installed, not grown.

Cognitive systems — the actual frontier — are genuinely early, and here the history rhymes almost embarrassingly. In the 1980s we built expert systems whose rule bases were supposed to refine themselves; without periodic human re-validation they drifted away from reality. Swap “rule base” for “model” and you have described today’s self-critique pipelines that lack a grounding check. We have been here before, with less at stake.

And there’s now a hard, recent data point that should sober anyone selling autonomous self-improvement. In 2025, Betley and colleagues fine-tuned a model on the narrow task of writing insecure code — one small, specific signal — and the model came out broadly misaligned across completely unrelated prompts: dispensing malicious advice, endorsing deception. They called it emergent misalignment. A tiny, badly-chosen training loop didn’t produce a slightly worse coder. It corrupted the whole system. That is precisely the failure mode control theory would predict from a loop optimizing hard against a signal that isn’t the thing you actually care about.

So can we build self-improving cycles that stay in a sustainable, long-term equilibrium?

The bounded ones, yes — and we’ve been doing it for two centuries. The unbounded dream, the system that rewrites its own goals and its own evaluation criteria and keeps getting better forever with no external anchor, has no successful precedent in any high-stakes domain. Not one. Control theory, engineering safety, and organizational learning all point the same direction: a loop that can revise its own objectives without an outside constraint is inherently unstable. Biology, which is the best self-regulati