KEP-6232keps/sig-node/6232-memory-manager-drift-tolerance
Tolerate benign per-NUMA memory drift in the Memory Manager
The Memory Manager Static policy records the total memory of each NUMA node in its checkpoint and, on kubelet start, requires the recorded per-node totals to match the current machine exactly. That total (cAdvisor's /sys/.../nodeN/meminfo MemTotal) is not stable across reboots, so kubelet crash-loops on a benign change with the expected machine state is diff… Full summary
The Memory Manager Static policy records the total memory of each NUMA node in its checkpoint and, on kubelet start, requires the recorded per-node totals to match the current machine exactly. That total (cAdvisor's /sys/.../nodeN/meminfo MemTotal) is not stable across reboots, so kubelet crash-loops on a benign change with the expected machine state is different from the real one, and the node stays NotReady until an operator deletes the checkpoint by hand. This KEP makes the policy tolerate a bounded, benign per-node memory drift and re-baseline onto the current machine, while still failing on genuine hardware or configuration changes. The tolerance is off by default and is turned on with a memory manager policy option, which derives the bound from the size of the running kernel image or pins it to a value. That shape follows the other kubelet policy options: the existing behavior stays the default until an administrator asks for the new one, and the option is also the only way to set the bound on a platform where it cannot be derived. Whether auto becomes the default once the feature is on by default is a beta decision, taken on the drift data gathered during alpha. The mechanism is Linux-specific: the drift comes from Linux kernel behavior and the bound is read from /proc/iomem. It is validated on x86-64, where all the reports so far come from. arm64 is expected to behave the same (the kernel image is placed randomly by the EFI stub and /proc/iomem carries the same entries) and will be confirmed during alpha. On every other platform, including Windows, the memory manager keeps its exact comparison and this feature is a no-op.
Facts
- Authors
- @ai-armless
- Approvers
- @mrunalp
- PRR approvers
- @johnbelamaricalpha
- Tracking issue
- #6232(opens GitHub in a new tab) openmilestone v1.38sig/nodestage/alphatracked/yeslead-opted-in
- Milestones
- alpha v1.38beta v1.39stable v1.41
- Participating SIGs
- none
- Feature gates
- MemoryManagerDriftTolerance
- Created / last updated
- 2026-07-13 / none