Possible memory leak when repeatedly creating controller-runtime clients: rest.Transport / tlsTransportCache grows unbounded

[i332322]

Memory growth observed when repeatedly creating controller-runtime clients

We are investigating a steady memory growth in a long-running controller and have narrowed it down to repeated creation of controller-runtime clients via client.New() on every reconcile.

After extensive profiling and controlled experiments, the memory growth appears to originate from rest.Transport / tlsTransportCache in client-go.

Environment

• Go version: 1.24.4
• controller-runtime: sigs.k8s.io/controller-runtime v0.22.4
• client-go: k8s.io/client-go v0.34.2
• Kubernetes version: v1.33.5
• OS/Arch: Linux amd64
• Controller: running in-cluster, long-lived, dynamically creates remote clients per reconcile

Suspect

remoteClient, err := client.New(remoteRestConfig, client.Options{
    Scheme: c.Scheme(),
    Mapper: c.RESTMapper(),
})

Observations

• Memory grows steadily over time even after GC runs.
• Heap profiles show retained allocations under rest.Transport and tlsTransportCache.
• Disabling client creation (e.g., invalid TLS config) stops the leak.

Question

Is it expected that clients created via client.New() retain transports indefinitely?

If so:

• Is there a recommended way to explicitly close or reuse transports?
• Should clients be cached/reused instead of recreated per reconcile?

Files included:

• heap.pprof

heap_20260105_135825_8n6kg.prof.txt
heap_20260107_094514_8n6kg.prof.txt

Profiles show retained memory growth originating from repeated calls to
controller-runtime client.New() and allocations under
rest.Transport / tlsTransportCache.

[alvaroaleman]

Is it expected that clients created via client.New() retain transports indefinitely?

Whatever happens there is inherited from client-go I believe. Is each of these clients completely unique in one or more of: target address, port, CA or client cert?

I would generally expect the cache to de-duplicate if possible. I don't know if it has a TTL mechanism, something like that is unlikely to be needed in upstream

[sbueringer]

Maybe unrelated but I think controller-runtime itself tries to re-use ~ a single HTTP client whenever possible (we pass it around in a few places)

(but of course not for a client.New() call)

[sbueringer]

Not sure if helpful for you, in Cluster API we cache clients with a component we call ClusterCache: https://github.com/kubernetes-sigs/cluster-api/tree/f871ce490d9a6dfdd79330608b200c180933316f/controllers/clustercache

In our case it's necessary because these clients are also using a backing cache and it would be unacceptable to create informers etc on every reconcile