Which proxy

It is Proxy(lconn, rconn io.ReadWriteCloser) error in pkg/daemons/control/proxy/proxy.go. It copies bytes both ways between two connections, one goroutine and one 32 KiB buffer per direction. The caller is TunnelServer.serveConnect in pkg/daemons/control/tunnel.go. It answers HTTP CONNECT: dial the backend, hijack the client connection, call proxy.Proxy. It serves the apiserver egress selector, the tunnel the apiserver uses to reach kubelets (k3s docs) and, in the reporter’s setup, aggregated APIServices.

The symptom

The issue has a goroutine dump from that server. Every leaked goroutine had the same stack, parked in chan send; one showed as goroutine 17335244 [chan send, 467 minutes]:. The count grew by roughly 11 per hour. The server log showed lines like Proxy error: write failed: ... write: connection reset by peer. I did not find the leak; these numbers are the report’s.

Why it never returned

The code before the fix (lines 26-40):

	go p.pipe(p.lconn, p.rconn)
	go p.pipe(p.rconn, p.lconn)
	return <-p.errc
}

func (p *proxy) err(err error) {
	if p.done {
		return
	}
	if !errors.Is(err, io.EOF) {
		logrus.Warnf("Proxy error: %v", err)
	}
	p.done = true
	p.errc <- err
}

errc was unbuffered and Proxy receives from it once. done was meant to keep a second goroutine from sending, but it is a plain bool written by both goroutines. Both can pass if p.done before either sets it. The Warnf call sits in that gap.

The goroutine that leaks is not stuck in Read or Write. Those already failed. It is parked on p.errc <- err. The Go spec says “A send on an unbuffered channel can proceed if a receiver is ready.” Proxy has returned, so none ever is. The deferred Close calls wake a goroutine blocked in Read or Write, not one blocked on a channel. Per the PR description, it holds its buffer and both connections.

The fix

From the merged diff of pkg/daemons/control/proxy/proxy.go (excerpt):

-		errc:  make(chan error),
+		errc:  make(chan error, 2), // one send per pipe goroutine, so neither can block
 	}
 
 	defer p.rconn.Close()
 	defer p.lconn.Close()
 	go p.pipe(p.lconn, p.rconn)
 	go p.pipe(p.rconn, p.lconn)
-	return <-p.errc
-}
-
-func (p *proxy) err(err error) {
-	if p.done {
-		return
-	}
-	if !errors.Is(err, io.EOF) {
+	err := <-p.errc
+	if !isConnTeardown(err) {
 		logrus.Warnf("Proxy error: %v", err)
 	}
-	p.done = true
-	p.errc <- err
+	return err
+}
+
+// isConnTeardown reports errors that are expected when one side closes or resets the connection.
+func isConnTeardown(err error) bool {
+	return errors.Is(err, io.EOF) ||
+		errors.Is(err, net.ErrClosed) ||
+		errors.Is(err, syscall.ECONNRESET) ||
+		errors.Is(err, syscall.EPIPE)
 }

In pipe, p.err(errors.WithMessage(err, "read failed")) became p.errc <- err, and the same for the write. Two slots for two senders means no send can block, whatever Proxy does. The guard is gone, so the race is gone. I kept “return on the first error” instead of waiting for both directions as remotedialer does. Per the PR description, that way Proxy cannot hang on a connection whose Close does not unblock a pending Read.

What review changed

• @bheemareddy commented inline that the warning is noisy and should be debug, with a sketch that used a helper named isConnTeardown. Their own PR, #14810, took the remotedialer approach. They called this one “smaller fix and easily back portable”, and @brandond went with it.
• @brandond liked filtering ECONNRESET and EPIPE and suggested dropping the errors.WithMessage wrap. He gave no reason. His isConnTeardown used a type assertion, err.(syscall.Errno).
• I replied with errors.As, because a TCP error is not a bare syscall.Errno. That part was my suggestion, not the reviewer’s. On a loopback reset I got a *net.OpError, the assertion gave false, and errors.Is(err, syscall.ECONNRESET) gave true.
• Then brandond said his intent was to ignore normal closure, so these errors should not be logged even at debug. He pushed that and dropped the log-level test. The merged helper uses errors.Is, and it also keeps net.ErrClosed, which I had dropped.

How the test catches it

Test_UnitProxyBothDirectionsFail uses a failConn whose Read and Write always fail. Both directions fail together:

	conn := failConn{errors.New("i/o error")}
	before := runtime.NumGoroutine()
	for range 10000 {
		_ = Proxy(conn, conn)
	}
	g.Eventually(runtime.NumGoroutine, 5*time.Second).Should(gomega.BeNumerically("<=", before))

Eventually is there because the second goroutine exits just after Proxy returns. I ran this test against the old proxy.go. It failed five runs in five, leaving roughly 900 to 1,900 goroutines against a start of 2, depending on the machine, and -race printed WARNING: DATA RACE. Against the merged code, go test -race -count=20 passed. With GOMAXPROCS=1 the old code passed, as the PR warns, so run leak tests on more than one core.

A leak check you can copy

The old and new shape in miniature. It makes 10,000 calls on failing connections and counts goroutines:

package main

import (
	"errors"
	"fmt"
	"io"
	"os"
	"runtime"
	"time"
)

// failConn fails every Read and Write, like a connection that was reset.
type failConn struct{}

func (failConn) Read([]byte) (int, error)  { return 0, errors.New("reset") }
func (failConn) Write([]byte) (int, error) { return 0, errors.New("reset") }

func pipe(src, dst io.ReadWriter, report func(error)) {
	buf := make([]byte, 1<<15)
	for {
		n, err := src.Read(buf)
		if err == nil {
			_, err = dst.Write(buf[:n])
		}
		if err != nil {
			report(err)
			return
		}
	}
}

// proxy returns the first error from either direction. With old set it has
// the pre-fix shape: an unbuffered channel and a plain bool as the guard.
func proxy(old bool) error {
	a, b := failConn{}, failConn{}
	errc := make(chan error, 2)
	report := func(err error) { errc <- err }
	if old {
		errc = make(chan error)
		done := false
		report = func(err error) {
			if done {
				return
			}
			done = true
			errc <- err
		}
	}
	go pipe(a, b, report)
	go pipe(b, a, report)
	return <-errc
}

func main() {
	old := len(os.Args) > 1 && os.Args[1] == "old"
	before := runtime.NumGoroutine()
	for range 10000 {
		proxy(old)
	}
	time.Sleep(500 * time.Millisecond) // let goroutines that can exit, exit
	fmt.Printf("before=%d after=%d\n", before, runtime.NumGoroutine())
}

On Go 1.24.5, go run . new printed before=1 after=1 in 15 of 15 runs. go run . old left a different count every run: 83 to 117 in one batch of 15, 12 to 64 in another. Any count above 1 is the leak. My copy leaks less than the real test. I did not test why; it omits the logging and error wrapping. go run -race . old reports the race.

The lesson for io.Copy-style proxies

• Count senders and receivers on every channel, on every path. If the receiver may leave early, size the buffer for all senders.
• A plain bool shared by two goroutines is not a guard. Buffer the channel or use sync.Once.
• Test the failure path: connections that fail at once, many calls, then runtime.NumGoroutine. Use -race and several cores.

How others solve it

net/http/httputil.ReverseProxy handles protocol upgrades with the same shape (source): errc := make(chan error, 1), two copier goroutines, each sending exactly once. The parent reads one value, and a second only if the first was nil. A buffer of 1 is enough because the parent always reads at least once. The k3s buffer of 2 does not depend on that. The copiers also pass on a half-close with CloseWrite when the connection supports it.

rancher/remotedialer’s pipe uses no channel: a sync.WaitGroup, a closePipe that closes both ends, then wg.Wait() (source). It waits for both directions. #14810 followed it.

If you have chased a leak like this one, reply on X.

What the function does

It is the left-right consistency check for a disparity map. The header comment in modules/calib3d/include/opencv2/calib3d.hpp says:

validates disparity using the left-right check. The matrix “cost” should be computed by the stereo correspondence algorithm

StereoBM calls it after matching when disp12MaxDiff >= 0, with its disparity map and a CV_16S map of the best SAD score per pixel (modules/calib3d/src/stereobm.cpp, lines 1122-1123 and 1237). StereoSGBM does its own check inline, in stereosgbm.cpp from line 2147.

Disparities are CV_16S fixed point, 16 units per pixel. For each row, validateDisparity (stereosgbm.cpp, line 2514) makes two passes:

1. For each pixel x with disparity d, find where it lands in the other image, x2 = x - ((d + 8) >> 4). Per landing column, keep the disparity of the lowest-cost pixel. Two cols-long int arrays, disp2buf and disp2cost, hold this in one allocation.
2. For each pixel, round d down and up to two columns and look both up. If both entries exist and both differ from d by more than disp12MaxDiff, the pixel becomes invalid, which is (minDisparity - 1) * 16.

The first pass runs x from max(maxD, 0) to cols + min(minD, 0). A disparity inside [minDisparity*16, (minDisparity+numberOfDisparities)*16) always gives x2 in [0, cols). I checked 5,433,296 cases by brute force (columns 20, 50 and 160, minDisparity -6 to 6, 1 to 16 disparities). The bounds hold only if the data keeps that promise.

The input that broke it

A disparity outside that range, for example from a map computed with other parameters, or a negative one. Then x2 can go negative or reach cols, and the code indexed with it unchecked, in both the CV_16S and the CV_32S cost branches.

Two cases from my test, on a row of 160 columns. With minDisparity = 2 and numberOfDisparities = 16, pixel 159 with d = 2672 gives x2 = 159 - 167 = -8. With minDisparity = 0, pixel 159 with d = -32 gives x2 = 161.

The two directions fail differently. disp2cost starts at disp2buf + cols, so disp2cost[-8] is disp2buf[152], a live entry for other pixels: the write silently changes another pixel’s answer. Its partner write, disp2buf[-8], lands 32 bytes before the buffer. disp2cost[161] is 4 bytes past the end of the 320-int buffer.

The issue’s snippet uses random 100x100 maps with values 0 to 255, minDisparity = 3 and numberOfDisparities = 7. Enumerating that range, x2 runs from -6 to 99, so only the negative side is reachable. With 100 columns the buffer is 200 ints, which fits inline in AutoBuffer<int>, 264 by default (modules/core/include/opencv2/core/utility.hpp, line 100). The disp2cost[x2] writes land on slots 94 to 99 of the same array. The paired disp2buf[x2] writes land before the array: x2 of -4 and -3 overwrite the ptr field of the AutoBuffer object, and -2 and -1 overwrite sz. ~AutoBuffer calls delete[] ptr when ptr != buf, so a damaged ptr can end in a bad free. That would fit the reporter’s exit code, 0xC0000374 (heap corruption), but I could not test their MSVC build. I ran the snippet 100 times each on the 4.9.0 and 5.0.0 Python wheels, macOS arm64: all 200 runs ended with exit status 0. In the 4.9.0 wheel the compiled code keeps ptr in a register and never reloads it, so the damage goes unseen.

This line has failed before. PR #4190, merged in 2015, skipped d == INVALID_DISP_SCALED because “x2 could be out of bounds if, for instance, x==cols-1 and d==INVALID_DISP_SCALED”. Mine covers the other values.

The diff

Six added lines, the same three in each branch:

                 int x2 = x - ((d + DISP_SCALE/2) >> DISP_SHIFT);
 
+                if( (unsigned)x2 >= (unsigned)cols )
+                    continue;
+
                 if( disp2cost[x2] > c )

Casting to unsigned turns a negative index into a huge one, so one comparison covers both ends. A skipped pixel does not vote in the reverse map, and is not itself changed: in the test it keeps its input value.

How the test proves it

Calib3d_ValidateDisparity.out_of_range_disparity runs one 160-column row for CV_16S and CV_32S, in both directions: four cases. It compares the whole row with the expected one. The negative case also shows up without a sanitizer:

pixels.push_back({154, 32, 0, 32});
pixels.push_back({155, 48, 10, invalid});
pixels.push_back({159, 2672, 0, 2672});

Pixels 154 and 155 both map to slot 152, and 154 is cheaper, so 155 fails the cross-check. Without the fix, pixel 159 overwrites slot 152 and 155 is not invalidated. Per the PR, without the fix all four cases fail under ASan, and the two negative ones also fail without a sanitizer. With the fix all four pass, and the existing StereoBM and StereoSGBM tests pass under ASan.

The work buffer is on the heap only above 132 columns (2 x 133 = 266 > 264). A narrower row sits inside the AutoBuffer object, where ASan has no redzone.

The earlier attempts and the review

#28507 was closed by its author the day it opened: “Closing this PR in favor of a clean re-submission.” #28508 was closed in March 2026, and its thread gives no reason. Before that, @asmorkalov saw no crash on Linux, with or without ASan, and asked for a test that reaches the guarded index.

On #30316, @asmorkalov approved with no text and merged. Right after the merge he asked: “Sounds like the same issue exists in 5.x. Could you submit complementary PR there?” It does: the 5.x copy in modules/stereo/src/stereosgbm.cpp had the same unguarded lines. I sent the port, with the same test, as #30344.

Run it

The wheels I tried have no fix. With the fix, the PR’s test expects pixel 155 to be 16 in both columns; I did not rebuild OpenCV to print that.

import subprocess
import sys

import cv2
import numpy as np

COLS, MIN_D = 160, 2
INVALID = (MIN_D - 1) * 16  # "no disparity" marker


def pixel_155(bad, dtype):
    disp = np.full((1, COLS), INVALID, np.int16)
    cost = np.zeros((1, COLS), dtype)
    disp[0, 154] = 32                    # 154 and 155 both map to column 152
    disp[0, 155], cost[0, 155] = 48, 10  # 154 is cheaper, so 155 must lose
    if bad:
        disp[0, 159] = 2672              # valid range is [32, 288)
    return int(cv2.validateDisparity(disp, cost, MIN_D, 16, 0)[0, 155])


for dt in (np.int16, np.int32):
    print(np.dtype(dt).name, pixel_155(False, dt), pixel_155(True, dt))

# x2 past the last column, in a child process
child = (
    "import cv2, numpy as np\n"
    "disp = np.full((1, 160), -16, np.int16); disp[0, 159] = -32\n"
    "cv2.validateDisparity(disp, np.zeros((1, 160), np.int16), 0, 16, 0)\n"
)
print("exit status:", subprocess.run([sys.executable, "-c", child]).returncode)

On 4.9.0 and on 5.0.0 it printed:

int16 16 48
int32 16 48
exit status: 0

The two numbers are pixel 155 without and with the far-off pixel; it should be 16. The past-the-end case exits with status 0 and no message.

The lesson

If a per-pixel loop turns a value from the map into an index, check the index where you use it. The loop bounds came from parameters, and parameters do not constrain the data. A map comes from another stage with other settings. Its invalid marker may fall outside this call’s range, and only this call’s own marker is skipped.

Make the test fail first: run it against the unfixed code. Mine needed 160 columns and a pixel whose result another pixel can change.

For how others solve it, look at the neighbours. The final loop of this same function range-checks x0 and x1, and the StereoSGBM inline check does the same with 0 <= _x && _x < width. The 2015 fix and this one both check at the use site.

If you have hit a disparity or depth loop that read past a row, I would like to hear how you found it. Reply on X.

The first result was not about the domain gap. A public checkpoint scored zero exact boards on my renders, and I read that as sim-to-real shift. About two thirds of the wrong squares came from a colour convention, and most boards still fail after the fix. Changing one renderer expression moved per-square accuracy from 54.4 to 85.2. The code and receipts are in the GitHub repo. The development renders are in the Hugging Face dataset. The dataset holds the 16,000 development images. The 4,000 sealed images are not released, and the 192 probe renders from the earlier lineage are not published, only their receipts.

The setup

The generator draws a procedural board, a piece family, a perspective camera, and explicit lighting and materials. Labels come from the scene description, never from a detector. A label is a piece-placement FEN, which names all 64 squares. The model under test is the public ChessQueries checkpoint. Its weights are released, so I could score it as is.

What the gap looked like

My early renders came in 18 splits from one generator lineage, with one board colour convention throughout. On every split the checkpoint produced zero exact boards. Per-square accuracy was 52.7 to 56.7. That is 4.6 to 9.0 points below a constant all-empty prediction, which also gets zero exact boards on these positions. So the model was not defaulting to empty. It was putting pieces on the wrong squares.

The same checkpoint, through the same code path, scored 99.62 exact on ChessReD (2,129 real photographs) and 98.54 on ChessCog, which is also rendered. The failure looked specific to my renders.

I tried the usual input fixes. A crop diagnostic fed the model 450 boards, 150 from each of three splits, four ways: full frame, bounding-box crop, square-padded crop, and a perspective warp to a square. None produced an exact board. Per-square accuracy moved by at most 1.2 points from the full frame. My notes at that point read: “the shift is the piece and board appearance itself.”

Why I believed it

The model was good on photos and on another rendered set, and bad on mine. Cropping and warping did nothing. What was left was appearance, which is the textbook story for a sim-to-real gap. You can tell it without verifying anything.

What the convention audit recorded

I recorded the board colour convention of one representative image from each of 56 source directories (some are retry or copy lineages). I use a1_light for a1 and h8 light, a8 and h1 dark, and a1_dark for the reverse. The laws of chess put a dark square at a1. My renderer, following the convention of the data it had been matched to, made a1 light.

In the sampled image, ChessReD, ChessCog, CVChess and ChessRender360 have a8 and h1 light, so a1 is dark. The sampled SLCC frame has no printed coordinates, so I recorded it as unresolved. My own procedural sources were a1 light in every sampled record. I also read the reviewed loaders. None of them mirrors the labels to compensate for board colour.

The one-expression fix

The old renderer coloured a square from the parity of its row and column: (r+c)%2===1?lm:dm. Here lm is the light material and dm the dark one. The row r counts ranks in FEN order, so rank 8 is row 0, and c counts files from a. Square a1 is row 7, column 0. Seven is odd, so a1 got lm. Swapping the branches makes it dark. The probe applies the swap as a patch to the renderer source. The pattern tolerates whitespace, and the patch refuses to run unless it matches exactly once:

def swap_parity(source):
    pattern = rb"(\(r\+c\)%2===1\s*\?\s*)lm(\s*:\s*)dm(?![A-Za-z_0-9])"
    if len(re.findall(pattern, source)) != 1:
        raise ValueError("expected exactly one original checkerboard assignment")
    return re.sub(pattern, lambda match: match[1] + b"dm" + match[2] + b"lm", source)

I rendered 192 positions with and without the patch. The control renders had to be byte-identical to the frozen originals, or the run stopped. Position, camera, piece geometry and projected corners had to match inside each pair, and the two images had to differ. The patch changes the physical checkerboard colours. It does not touch the label order or the image orientation.

Before and after

The released checkpoint on 192 positions:

The swap removed 67.65% of the wrong squares. The 95% paired bootstrap interval on the +30.88 per-square gain is 28.91 to 32.76. On 288 positions from six other piece families, the swap took exact boards from 0/288 to 50/288 and per-square from 55.31 to 87.49.

The checkpoint carries a learned two-class colour index for its 64 square queries. Flipping it in memory on the original renders recovers 20.7 points but no exact board, and costs 2.9 points on the swapped ones. That fits a colour prior contributing to the failure. It does not isolate the mechanism.

Most boards still fail after the fix. Only 18 of 192 are exact, and in the lowest of four elevation bands none of 48 are. The convention explained about two thirds of the wrong squares. The rest is unexplained. It may be a real domain gap, or camera elevation, framing and piece geometry. This probe cannot say how much.

On the 400-group development partition of the v2 benchmark, the same checkpoint scores 15.48 exact pooled under a1 dark and 0.0 in all twenty cells under a1 light.

The lesson

Check label and convention alignment before you blame the domain gap. Four checks are cheap.

• Write down which corner is light in your data and in the data the model saw.
• Read the loaders for any reflection of images or labels.
• Treat a score below a constant baseline as a mapping error until shown otherwise. That is my rule of thumb, not a result from the repo.
• Make the convention a factor and render both. In v2 it is the fifth axis, so every scene exists in both conventions.

What else the benchmark measures

The design has 500 position groups, 400 for development and 100 sealed. Five factors vary one at a time: camera elevation, piece geometry, board palette, framing, and the colour convention. Twenty factor cells at both conventions give 20,000 images, 18,000 of them unique renders. Positions are disjoint from 67 inventoried sources, exactly and under the eight board symmetries. “Sealed” is my no-scoring rule, not an access control. I scored no model on that split.

Three more findings:

• Low-rank adaptation on 500 rendered positions lifted the checkpoint from 15.5 to 64.7 exact on the standard convention (dark-only training), and 8,000 positions reached 79.7 (one seed each). Every one of ten adapted checkpoints then lost 11 to 23 boards on the 2,129 real photos, against a retention gate fixed before training.
• The lowest-elevation a1-dark cell reaches only 14.5 exact after the largest both-convention budget. The reference-elevation cell reaches 86.5.
• A pipeline selected on a private leaderboard scored below its own single member on 16 of 18 controlled splits. Two sets named as holdouts shared 1,446 and 1,128 image identifiers with the training mixes.

If a convention mismatch ever passed as a domain gap in your pipeline, I would like to hear how you found it. Reply on X.

This post covers why those five lines cost so much and why they could go.

What network-policy sync does

kube-router’s network policy controller turns Kubernetes NetworkPolicy objects into iptables rules on every node. k3s embeds it: the k3s docs say “K3s includes an embedded network policy controller. The underlying implementation is kube-router’s netpol controller library”. k3s pins its own fork of kube-router in go.mod (v2.6.3-k3s1), and that tag has the same line I removed. The fix is on kube-router master. The latest release I can see as of 2026-10-15, v2.11.1 from August, predates the merge. So k3s clusters get it only after that pin moves.

The controller works by full sync. fullPolicySync does roughly this:

1. Run iptables-save for the filter table into a bytes.Buffer, one per IP family.
2. Append the policy chains (KUBE-NWPLCY-*) and, for every pod on the node, a firewall chain (KUBE-POD-FW-*) with its rules.
3. Strip stale rules from the buffer.
4. Hand the buffer to iptables-restore.

A pod event calls RequestFullSync, and so does a periodic timer. Requests go through a channel of capacity 1, and syncs run one at a time. The issue behind the PR (#2191) reports CPU profiles where fullPolicySync took 26-58% of each node’s k3s process CPU (the whole function, not only this check).

The check

Inside syncPodFirewallChains, for every local pod and each IP family the pod has an address in, this ran before the pod’s log rule was written:

// This used to be AppendUnique when we were using iptables directly, this checks to make sure we didn't drop
// unmarked for this chain already
if strings.Contains(filterTableRules.String(), strings.Join(args, " ")) {
	continue
}

The comment is the history. When kube-router drove iptables one command at a time, AppendUnique meant “add this rule unless it is already there”. Once the rules were built in a buffer, the check became a search of the buffer.

Why it cost so much

bytes.Buffer.String() ends in one line in the Go standard library:

return string(b.buf[b.off:])

That is a copy of the whole buffer. At this point the buffer holds the full iptables-save output plus every rule generated so far. Then strings.Contains scans the copy. The rule never matched, so every scan ran to the end.

So each pod cost one allocation, one copy and one scan of the entire table. Total work is pods times table size. The table also grows with pods, because each pod adds a chain and its rules. In the benchmark, 80 copies of a 1 MiB buffer is 83,886,080 bytes. The measured 90,489,076 B/op is close to that.

Using bytes.Contains(buf.Bytes(), ...) would have removed the copy, since Bytes() returns a slice that aliases the buffer. It would not have removed the scan. The cost would still scale with pods times size.

The fix, in two steps

My first version replaced the buffer search with a per-family set of the rules written in this sync:

logRule := strings.Join(args, " ")
if loggedDrops[ipFamily][logRule] {
	continue
}
loggedDrops[ipFamily][logRule] = true

On the benchmark it gave the same result as the final version. Then the maintainer, @aauren, reviewed it. The review comment said they would argue “we should just drop this check altogether”, because “as far as I can tell this is dead code and never has the potential of actually coming back with a match, unless you see it differently?”

That argument holds up. The rule text contains:

• the pod’s chain name, which is KUBE-POD-FW- plus 16 characters of base32(sha256(namespace + podName + version));
• a <namespace>/<name> log-drop comment.

The version is derived from the sync’s start time in nanoseconds, so rules left by an earlier sync carry a different chain name. For a match inside one sync, the same pod would have to come through the loop twice in the same family. The maintainer’s read was that a real cluster cannot do that. I could not find a way either.

My description had already said that rules from an earlier sync could never match, because the chain name contains the sync version and iptables-save prints the rule differently (10/min instead of 10/minute, and a different match order). My set was guarding against a case that could not happen. I agreed, deleted the check and the set, and dropped the two tests that only covered them. The final diff in pod.go is five deletions and no additions. @aauren approved it and merged it on 2026-10-12.

A side effect: while checking whether pods with colliding chain names could trigger the check, I noticed that the hash joins namespace and name with no separator. Pods a/bc and ab/c get the same chain name. That is a separate bug, and the maintainer filed it as #2193, which covers more than pod chains. I left it alone in this PR. The log-drop comment still differs for those two pods, so the old check did not fire for them either.

How I measured

First, a test to pin behaviour. TestSyncPodFirewallChainsDropRules asserts the exact rules written per chain and per family, for single-stack, dual-stack and colliding-name pods. The PR says it passes on master and on the branch. A change that deletes a check needs a test that agrees with the old code.

Then the benchmark. It builds 80 pods in one IP family and fills a buffer with synthetic policy-chain lines past 1 MiB, as if iptables-save had just run:

for i := 0; existing.Len() < 1<<20; i++ {
	fmt.Fprintf(&existing, "-A KUBE-NWPLCY-%016X -m comment --comment \"ns-%d/policy-%d ingress pods\" "+
		"-m set --match-set KUBE-SRC-%016X src -j MARK --set-xmark 0x10000/0x10000\n", i, i%40, i, i)
}

for b.Loop() {
	buf.Reset()
	buf.Write(existing.Bytes())
	npc.syncPodFirewallChains(nil, "bench")
}

The numbers are the median of -count 5. Allocations per op went from 2115 to 1954, which is 161 fewer. That fits two fewer per pod, the String() copy and one strings.Join.

Some limits. The loop also resets and refills the 1 MiB buffer on each pass, so that time is inside both numbers, and 477,097 ns/op is an upper bound for the function, so the 320x ratio is understated. The benchmark covers one function, not a whole sync. I did no cluster testing for this PR. The issue’s own standalone benchmark, at 84 pods and a 1.2 MB buffer, measured 180-188 ms per sync for the check, which is the same order as mine.

What I take from it

• A call that looks like a read can allocate. Inside a loop, ask what it costs per iteration.
• Before making a check faster, ask whether it can ever be true. I built a set. The maintainer asked whether the check was needed at all, which was the better answer.
• Pin behaviour with a test that passes before and after, then change the code.

How others solve it

kube-proxy in iptables mode has the same shape: build the rules in line buffers, then restore them with NoFlushTables. I found no search of those buffers in it. It tracks the chains it wants in a set (activeNATChains := sets.New[utiliptables.Chain]()). It reads iptables-save only to list chain names with GetChainsFromTable and diffs them against that set. In large clusters it does that at most once per sync period, because, as its own comment says, iptables-save “can take several seconds to run on hosts with lots of iptables rules”. It also has a doFullSync flag, so not every sync rebuilds everything.

Keep the state in a data structure, and do not search the output buffer once per item.

If you run kube-router or k3s with many pods per node, I would like to hear what your sync times look like. Reply on X.

The bug is one Python habit, not one Redis quirk. You may have it in your own code.

The symptom

Turn the cyclic garbage collector off. Send 1,000 error replies through the asyncio client. Count how many ResponseError objects are still alive, using weakrefs. Before the fix: 1000 of 1000. After: 0 of 1000.

I ran this against a live server with two replies: NOSCRIPT from EVALSHA, and WRONGTYPE from LPUSH on a string. Both gave 1000 / 1000 before and 0 / 1000 after. The sync client already gave 0 / 1000.

The objects are freed once the collector runs. The question is how long that takes, and what else they hold in the meantime. The PR lists replies that hit this: NOSCRIPT from EVALSHA (which AsyncScript retries on), WRONGTYPE, BUSYGROUP, and MOVED/ASK in cluster mode.

One local is the whole bug

This is the code that was there:

if isinstance(response, ResponseError):
    raise response from None
return response

response is a local variable. When raise runs, the exception gets a __traceback__, and the traceback points at every frame between where it was raised and where it is caught. One of them is the read_response frame. That frame owns its locals, and one local is response, the exception itself.

So: exception, __traceback__, frame, locals, exception. Reference counts on a loop never reach zero. Only the cyclic collector can free it, and until it does, every frame on the traceback stays alive with its locals.

Why except ... as e does not do this

You write except Boom as e: all the time and do not leak. Python deletes e for you. From the language reference:

When an exception has been assigned using as target, it is cleared at the end of the except clause. This is as if:

except E as N:
    foo

was translated to:

except E as N:
    try:
        foo
    finally:
        del N

The docs give the reason: “Exceptions are cleared because with the traceback attached to them, they form a reference cycle with the stack frame, keeping all locals in that frame alive until the next garbage collection occurs.” PEP 3110 added that cleanup in Python 3.0.

The cleanup covers one name. Any other name bound to the exception is yours to clear: a local you raise, like response, or a copy such as saved = e.

Why it matters

The collector frees these cycles eventually. But the gc docs say: “you can disable the collector if you are sure your program does not create reference cycles.” Their recipe for a process that calls fork() without exec() is gc.disable() early in the parent, gc.freeze() right before fork(), gc.enable() early in the children. Instagram Engineering has written about dismissing the collector entirely: Dismissing Python Garbage Collection at Instagram.

An app that made that bet does not expect a library to make one cycle per error reply. Raise the thresholds, or turn the collector off, and each cycle is held longer, or for good.

The fix

if isinstance(response, ResponseError):
    try:
        raise response from None
    finally:
        del response  # avoid creating ref cycles
return response

You cannot del response before the raise, because the raise needs it. The finally block runs while the exception is already in flight, so the delete comes after the raise has picked the object up. The frame no longer points at the exception. The exception still points at the frame, but a one-way link is not a cycle. When the caller drops the exception, the count hits zero and everything is freed on the spot. from None stays as it was: it only hides the implicit __context__ in the traceback display and plays no part in the cycle.

The sync client has had the same pattern since #2932 (merged 2023-09-14), with the same comment.

How the test proves it

test_read_response_error_reply_breaks_reference_cycle needs no server. It swaps in a parser stub that returns a ResponseError:

    error_refs = []

    async def error_reply(*args, **kwargs):
        error = ResponseError("NOSCRIPT No matching script. Please use EVAL.")
        error_refs.append(weakref.ref(error))
        return error

    conn._read_response_from_parser = error_reply

    gc.collect()
    gc_was_enabled = gc.isenabled()
    gc.disable()
    try:
        with pytest.raises(ResponseError):
            await conn.read_response()

        (ref,) = error_refs
        assert ref() is None

(Excerpt. conn = Connection() comes first, and a finally re-enables the collector if it was on.)

Three details make it honest. The error is built inside the stub, so the weakref is the only handle the test has. pytest.raises has no as excinfo, so the test does not hold the exception either. I checked: bind excinfo and the weakref stays alive. And the collector is off, so only reference counting can free it. If a cycle exists, ref() is not None. The PR says the test fails without the change and passes with it on Python 3.10, 3.11 and 3.14. It is skipped on PyPy, with the reason “Immediate reclamation is a refcounting property”.

I checked it a second way. I ran the same stub, 1,000 calls, against redis-py 8.1.0 from PyPI: 1000 / 1000 alive. With the four-line change applied to a copy: 0 / 1000.

Spot it in your own code

Run a suspect function many times with the collector off, and ask what only a GC pass could free. gc.DEBUG_SAVEALL keeps collected objects in gc.garbage, so you can read their types:

import gc


class Boom(Exception):
    pass


def leaky():
    response = Boom("nope")
    raise response


def fixed():
    response = Boom("nope")
    try:
        raise response
    finally:
        del response


def with_as():
    try:
        raise Boom("nope")
    except Boom as e:
        pass


def saved_as():
    try:
        raise Boom("nope")
    except Boom as e:
        saved = e


def left_for_gc(fn, calls=1000):
    gc.collect()
    gc.disable()
    gc.set_debug(gc.DEBUG_SAVEALL)  # collected objects go to gc.garbage
    try:
        for _ in range(calls):
            try:
                fn()
            except Boom:
                pass
        gc.collect()
        errors = [o for o in gc.garbage if isinstance(o, Boom)]
        kinds = sorted({type(o).__name__ for o in gc.garbage})
        return len(errors), kinds
    finally:
        gc.set_debug(0)
        gc.garbage.clear()
        gc.enable()


for fn in (leaky, fixed, with_as, saved_as):
    n, kinds = left_for_gc(fn)
    print(f"{fn.__name__:8} {n:4} exceptions left for the GC  {kinds}")

On Python 3.14.7 it prints:

leaky    1000 exceptions left for the GC  ['Boom', 'frame', 'traceback']
fixed       0 exceptions left for the GC  []
with_as     0 exceptions left for the GC  []
saved_as 1000 exceptions left for the GC  ['Boom', 'frame', 'traceback']

On 3.14 the three type names are the cycle: the exception, its traceback and the frame. Python 3.13 and older also list a tuple, the exception’s args. with_as is clean because Python deletes e. saved_as leaks because saved is a second name.

How others solve it

The language solves it for except ... as by deleting the name. The standard library solves the other cases by hand. This is Future.__get_result in concurrent/futures/_base.py from Python 3.14.7:

        if self._exception is not None:
            try:
                raise self._exception
            finally:
                # Break a reference cycle with the exception in self._exception
                self = None

Same shape: raise inside try, clear the name in finally. There the loop runs through self, so it clears self instead of the exception.

A related PR, #4439, was still open on 2026-10-15. It targets the same kind of cycle in NodesManager.initialize, where exception = e stays bound after a later startup node answers.

If you have chased a reference cycle like this one, tell me how you found it. Reply on X.

None of the four problems below failed a test. Each showed up elsewhere: a process table, a disk counter, a frozen terminal, a loop that never ended. Each became a rule checked before the command runs, not a note written after it breaks.

1. Zombies from containers without an init

What happened. One container, started by an agent with a plain docker run, ended up with 790 defunct git processes. They used no CPU, but they filled the process table and the count kept climbing.

Why. A zombie is a child that has exited while its parent has not called wait() to collect its exit status. When a parent dies, its children are reparented to PID 1, and on a normal host PID 1 is an init that reaps them. Inside a container, PID 1 is whatever your entrypoint is, and most programs were never written to do init’s job. An agent’s shell spawns git, a timeout kills the shell, the orphan lands on PID 1, and nobody reaps it. You cannot kill a zombie, because it is already dead. You can only get its parent to reap it, or end the parent.

The fix. Run containers with an init:

docker run --init --rm --pids-limit 512 my-image my-command

--init makes Docker run a tiny init as PID 1 that starts your command, forwards signals, and reaps orphans. In Compose it is init: true. --pids-limit is a seatbelt: a leak hits a ceiling instead of taking the host. Find zombies with:

ps -eo stat,ppid,args | awk '$1 ~ /^Z/'

How others solve it. Docker’s --init is built on tini, and you can put tini or dumb-init in the image with ENTRYPOINT ["tini", "--"]. That is the portable answer, since Kubernetes has no --init field on a container. Setting shareProcessNamespace: true on a pod makes its pause process PID 1 for every container, and it reaps orphans, at the price of containers seeing each other’s processes. Init containers run once before the app starts and reap nothing.

2. A test database that wore down an SSD

What happened. Agents kept starting a throwaway Postgres to test against. For about a week it wrote roughly 1.1 TB a day to an SSD. Every test passed, so nothing prompted anyone to look.

Why. That is only about 13 MB/s on average, which looks unremarkable on a graph, but it is around 400 TB a year, more than many consumer drives are rated for. Postgres is built to survive a crash: every commit flushes the write-ahead log with fsync, and the first change to a page after a checkpoint writes the whole page to the log (full_page_writes). A suite committing thousands of tiny transactions pays that cost every time, to protect data deleted a minute later.

The fix. Keep test databases in RAM and turn off the durability you do not need:

docker run --init --rm -p 127.0.0.1::5432 \
  --tmpfs /var/lib/postgresql/data:size=1g \
  -e POSTGRES_PASSWORD=test \
  postgres:17 \
  -c fsync=off -c synchronous_commit=off -c full_page_writes=off

On the official Postgres 18 images the data directory moved, so mount the tmpfs at /var/lib/postgresql instead. Binding to 127.0.0.1 keeps the database off the network; the empty host port lets Docker pick one (docker port shows it). The Postgres documentation is blunt that fsync=off can corrupt a cluster after a crash. That is fine here: on tmpfs a crash loses the database anyway.

To catch a heavy writer early, iostat -dx 5 shows write throughput per device and pidstat -d 5 per process. For the long view, smartctl -A on an NVMe drive reports “Data Units Written” with a terabyte figure beside it. Record it daily and diff it.

How others solve it. CI systems commonly run service databases on tmpfs for the same reason. Many suites also roll back a transaction per test, or clone a template database with CREATE DATABASE ... TEMPLATE, so there is less to write.

3. One benchmark froze a shared box

What happened. A benchmark an agent launched used about 35 GB on a machine without that much to spare. There was no tidy out-of-memory kill. The machine froze for everyone on it, including services unrelated to the benchmark. A timeout in an agent’s tool layer does not help: it can abandon a command without killing it.

Why. Linux does not act the moment memory runs out. It reclaims page cache, then swaps, and a box thrashing swap can be unusable for a long time before the OOM killer picks a victim. The victim is chosen system-wide, not necessarily the process that caused the problem. A cgroup limit changes the scope: when a cgroup hits its limit, the OOM killer runs inside that cgroup.

The fix. Run heavy jobs in a transient systemd scope with a hard cap:

systemd-run --user --scope --collect \
  -p MemoryMax=8G -p MemorySwapMax=0 -- ./run-benchmark

MemoryMax is the hard limit; per the systemd docs, the OOM killer is invoked inside the unit when usage cannot be held under it. MemorySwapMax=0 keeps the job out of swap, so it dies quickly instead of dragging the machine down. --collect unloads the unit even if it failed, so failed scopes do not pile up. Pick the smallest cap that does the job. On a cgroup v2 distro this normally works as a user; if not, use sudo for a system scope. MemoryHigh is the softer option: it throttles instead of killing.

How others solve it. Docker has --memory=8g --memory-swap=8g; equal values mean no swap. Kubernetes has resources.limits.memory, and a container over it is OOM-killed. Underneath, it is all the cgroup v2 files memory.max, memory.high and memory.swap.max. A host-wide backstop like systemd-oomd or earlyoom helps, but a per-job cap keeps the failure small.

4. Waiting forever

What happened. Agents wait on things: a build, a deploy, a service coming up. Their default is a loop of sleep, check, sleep, check. The loop had an exit for success and none for anything else, so a job that had failed, or never started, was polled until I noticed. Every check was also a model call, so waiting cost money too.

Why. The loop encodes one hope. Nothing in it says how long is too long, and chained short sleeps are the same loop.

The fix. Every wait gets a deadline and a growing delay, and waits on the event where possible:

wait_for() {   # bash
  local deadline=$((SECONDS + 180)) delay=2
  until "$@"; do
    [ "$SECONDS" -ge "$deadline" ] && { echo "gave up: $*" >&2; return 1; }
    sleep "$delay"
    delay=$(( delay * 2 > 30 ? 30 : delay * 2 ))
  done
}
wait_for curl -fsS http://localhost:8080/health

When the system can answer, ask it: kubectl wait --for=condition=Ready pod/my-pod --timeout=120s blocks until the condition holds or the timeout hits. timeout 240 <command> caps anything, and exit status 124 means the cap was hit.

How others solve it. Most infrastructure has deadlines built in: Kubernetes Jobs have activeDeadlineSeconds, CI systems have per-job timeouts, systemd has TimeoutStartSec. Retrying clients use exponential backoff, often with jitter so a crowd does not retry in step.

The common thread

Each of these is something a tired human could also have done. Agents do it faster, more often, with nobody watching the machine. So I stopped writing postmortems and started writing preconditions: the container has --init, the test database is on tmpfs, the heavy job has a memory cap, the wait has a deadline. Each is checked before the command runs.

If you run agents on machines you care about, I would like to hear what yours have done to them. What is the strangest resource burn you have caught? Reply on X.