KEP-2238keps/sig-node/2238-liveness-probe-grace-period

Liveness Probe Grace Period

implementablestablesig-nodelatest v1.281 open finding

KEP on GitHub kep.yaml Ask about this KEP

Liveness probes currently use the terminationGracePeriodSeconds on both normal shutdown and when probes fail. Hence, if a long termination period is set, and a liveness probe fails, a workload will not be promptly restarted because it will wait for the full termination period. I propose adding a new field to probes, probe.terminationGracePeriodSeconds. When set, it will override the terminationGracePeriodSeconds for liveness or startup termination, and will be ignored for readiness probes. This maintains the current behaviour if desired while providing configuration to address this unintended behaviour.

Facts

Authors: @ehashman
Approvers: @derekwaynecarr
Reviewers: @dims @derekwaynecarr @matthyx @smarterclayton
PRR approvers: @johnbelamaricalpha@johnbelamaricbeta@johnbelamaricstable
Tracking issue: #2238(opens GitHub in a new tab) closednot_plannedclosed 2024-06-15 01:33 UTCsig/nodestage/stablelifecycle/rotten
Milestones: alpha v1.21beta v1.22stable v1.28
Participating SIGs: none
Feature gates: ProbeTerminationGracePeriod
Created / last updated: 2021-01-07 / none

Who decided each state, with the evidence.
Decided	Decision	Deciders	Evidence
2021-02-08 22:02 UTC	prr_approval johnbelamaric gave the production readiness approval for KEP-2238 at alpha on 2021-02-08.	@johnbelamaric prr_approver	Document 2238.yaml(opens GitHub in a new tab)
2021-02-08 22:02 UTC	stage_graduation KEP-2238 set milestone.alpha to v1.21 in PR #2241, approved by derekwaynecarr, johnbelamaric; merged by automation (k8s-ci-robot) on 2021-02-08.	@ehashman author @johnbelamaric approve_command @derekwaynecarr approver @derekwaynecarr approve_command @derekwaynecarr lgtm @matthyx lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@e9b84b147ad2d0e8697cb114498763887325283c(opens GitHub in a new tab) StatusChange milestone.alpha ∅ → v1.21 (2021-02-08)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2241 Configurable probe termination grace periods(opens GitHub in a new tab) Review kubernetes/enhancements/pulls/reviews/585865808(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/775487025(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/756571921(opens GitHub in a new tab)
2021-02-08 22:02 UTC	stage_graduation KEP-2238 set milestone.beta to v1.22 in PR #2241, approved by derekwaynecarr, johnbelamaric; merged by automation (k8s-ci-robot) on 2021-02-08.	@ehashman author @johnbelamaric approve_command @derekwaynecarr approver @derekwaynecarr approve_command @derekwaynecarr lgtm @matthyx lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@e9b84b147ad2d0e8697cb114498763887325283c(opens GitHub in a new tab) StatusChange milestone.beta ∅ → v1.22 (2021-02-08)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2241 Configurable probe termination grace periods
2021-02-08 22:02 UTC	stage_graduation KEP-2238 set milestone.stable to v1.23 in PR #2241, approved by derekwaynecarr, johnbelamaric; merged by automation (k8s-ci-robot) on 2021-02-08.	@ehashman author @johnbelamaric approve_command @derekwaynecarr approver @derekwaynecarr approve_command @derekwaynecarr lgtm @matthyx lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@e9b84b147ad2d0e8697cb114498763887325283c(opens GitHub in a new tab) StatusChange milestone.stable ∅ → v1.23 (2021-02-08)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2241 Configurable probe termination grace periods
2021-02-08 22:02 UTC	stage_graduation KEP-2238 graduated to alpha in PR #2241, approved by derekwaynecarr, johnbelamaric; merged by automation (k8s-ci-robot) on 2021-02-08.	@ehashman author @johnbelamaric approve_command @derekwaynecarr approver @derekwaynecarr approve_command @derekwaynecarr lgtm @matthyx lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@e9b84b147ad2d0e8697cb114498763887325283c(opens GitHub in a new tab) StatusChange stage ∅ → alpha (2021-02-08)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2241 Configurable probe termination grace periods
2021-02-08 22:02 UTC	status_transition KEP-2238 status set to implementable in PR #2241, approved by derekwaynecarr, johnbelamaric; merged by automation (k8s-ci-robot) on 2021-02-08.	@ehashman author @johnbelamaric approve_command @derekwaynecarr approver @derekwaynecarr approve_command @derekwaynecarr lgtm @matthyx lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@e9b84b147ad2d0e8697cb114498763887325283c(opens GitHub in a new tab) StatusChange status ∅ → implementable (2021-02-08)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2241 Configurable probe termination grace periods
2021-04-19 22:59 UTC	prr_approval johnbelamaric gave the production readiness approval for KEP-2238 at beta on 2021-04-19.	@johnbelamaric prr_approver	Document 2238.yaml(opens GitHub in a new tab)
2021-05-13 20:21 UTC	milestone_retarget KEP-2238 latest-milestone moved from v1.21 to v1.22 in PR #2638, approved by derekwaynecarr; merged by automation (k8s-ci-robot) on 2021-05-13.	@ehashman author @derekwaynecarr approve_command @derekwaynecarr lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@488232d9f276e60b0d12d081cbc0ce218b8b6dde(opens GitHub in a new tab) StatusChange latest_milestone v1.21 → v1.22 (2021-04-19)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2638 Update probe grace period KEP to beta(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/840809420
2021-05-13 20:21 UTC	milestone_retarget KEP-2238 milestone.stable moved from v1.23 to v1.25 in PR #2638, approved by derekwaynecarr; merged by automation (k8s-ci-robot) on 2021-05-13.	@ehashman author @derekwaynecarr approve_command @derekwaynecarr lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@4f9ca828ef3bfa86adc9f2ece05fe2a55bc5788a(opens GitHub in a new tab) StatusChange milestone.stable v1.23 → v1.25 (2021-05-12)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2638 Update probe grace period KEP to beta(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/840809420
2021-05-13 20:21 UTC	stage_graduation KEP-2238 graduated to beta from alpha in PR #2638, approved by derekwaynecarr; merged by automation (k8s-ci-robot) on 2021-05-13.	@ehashman author @derekwaynecarr approve_command @derekwaynecarr lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@488232d9f276e60b0d12d081cbc0ce218b8b6dde(opens GitHub in a new tab) StatusChange stage alpha → beta (2021-04-19)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#2638 Update probe grace period KEP to beta(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/840809420
2022-06-16 19:03 UTC	milestone_retarget KEP-2238 latest-milestone moved from v1.22 to v1.25 in PR #3282, approved by derekwaynecarr; merged by automation (k8s-ci-robot) on 2022-06-16.	@ehashman author @derekwaynecarr approve_command @derekwaynecarr lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@60b8a6f72775a1ac956bef914e094fa7f6ecb2fd(opens GitHub in a new tab) StatusChange latest_milestone v1.22 → v1.25 (2022-04-20)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#3282 KEP-2238: Update implementation history and milestones(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/1158024747
2022-06-16 19:03 UTC	milestone_retarget KEP-2238 milestone.stable moved from v1.25 to v1.27 in PR #3282, approved by derekwaynecarr; merged by automation (k8s-ci-robot) on 2022-06-16.	@ehashman author @derekwaynecarr approve_command @derekwaynecarr lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@60b8a6f72775a1ac956bef914e094fa7f6ecb2fd(opens GitHub in a new tab) StatusChange milestone.stable v1.25 → v1.27 (2022-04-20)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#3282 KEP-2238: Update implementation history and milestones(opens GitHub in a new tab) Comment kubernetes/enhancements/issues/comments/1158024747
2023-02-02 22:07 UTC	prr_approval johnbelamaric gave the production readiness approval for KEP-2238 at stable on 2023-02-02.	@johnbelamaric prr_approver	Document 2238.yaml(opens GitHub in a new tab)
2023-02-08 19:36 UTC	milestone_retarget KEP-2238 latest-milestone moved from v1.25 to v1.27 in PR #3830, approved by mrunalp, sergeykanzhelev, dchen1107, johnbelamaric; merged by automation (k8s-ci-robot) on 2023-02-08.	@dchen1107 approve_command @dchen1107 lgtm @mrunalp approver @sergeykanzhelev approver @sergeykanzhelev lgtm @johnbelamaric approve_command @rphillips author @harche lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@d368351677e408d63d54f14602bf6c4ee9ee671e(opens GitHub in a new tab) StatusChange latest_milestone v1.25 → v1.27 (2023-02-02)
2023-02-08 19:36 UTC	stage_graduation KEP-2238 graduated to stable from beta in PR #3830, approved by mrunalp, sergeykanzhelev, dchen1107, johnbelamaric; merged by automation (k8s-ci-robot) on 2023-02-08.	@dchen1107 approve_command @dchen1107 lgtm @mrunalp approver @sergeykanzhelev approver @sergeykanzhelev lgtm @johnbelamaric approve_command @rphillips author @harche lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@d368351677e408d63d54f14602bf6c4ee9ee671e(opens GitHub in a new tab) StatusChange stage beta → stable (2023-02-02)
2023-06-09 19:02 UTC	milestone_retarget KEP-2238 latest-milestone moved from v1.27 to v1.28 in PR #3986, approved by mrunalp, sergeykanzhelev; merged by automation (k8s-ci-robot) on 2023-06-09.	@mrunalp approver @sergeykanzhelev approver @sergeykanzhelev approve_command @sergeykanzhelev lgtm @rphillips author @bart0sh lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@3dcef7101dc854d2030005dd4f000aec40f516cf(opens GitHub in a new tab) StatusChange latest_milestone v1.27 → v1.28 (2023-05-05)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#3986 bump probe termination to ga for 1.28 (missed 1.27)
2023-06-09 19:02 UTC	milestone_retarget KEP-2238 milestone.stable moved from v1.27 to v1.28 in PR #3986, approved by mrunalp, sergeykanzhelev; merged by automation (k8s-ci-robot) on 2023-06-09.	@mrunalp approver @sergeykanzhelev approver @sergeykanzhelev approve_command @sergeykanzhelev lgtm @rphillips author @bart0sh lgtm @k8s-ci-robot merger	Commit kubernetes/enhancements@3dcef7101dc854d2030005dd4f000aec40f516cf(opens GitHub in a new tab) StatusChange milestone.stable v1.27 → v1.28 (2023-05-05)(opens GitHub in a new tab) PullRequest kubernetes/enhancements#3986 bump probe termination to ga for 1.28 (missed 1.27)

Metadata, label and milestone changes, oldest first.Showing 1–10 of 82Rows

Status history
At	Kind	Field	Change	Actor	Evidence
2021-01-07 22:24 UTC	label	sig/node added	none → sig/node	@k8s-ci-robot	source(opens GitHub in a new tab)
2021-02-02 19:12 UTC	milestone	milestone milestoned	none → v1.21	@k8s-ci-robot	source

Relationships as stored, Person → KEP, from kep.yaml(opens GitHub in a new tab)
Person	Relationship
@ehashman GitHub(opens GitHub in a new tab)	—AUTHORED→
@dims GitHub(opens GitHub in a new tab)	—REVIEWS→
@derekwaynecarr GitHub(opens GitHub in a new tab)	—REVIEWS→
@matthyx

How they connect

Shortest paths in the graph between this KEP and its SIG or one of its people, up to four hops.

Impact of change

Rows returned by the tool
Kep	Depth	Reached	By hop	Open pull requests	Hop	Feature gates	Sigs	People
{ "label": "KEP", "key": "keps/sig-node/2238-liveness-probe-grace-period", "title": "Liveness Probe Grace Period", "url": "https://github.com/kubernetes/enhancements/tree/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period", "evidence_id": "wqcp6l675lzv", "number": 2238, "owning_sig": "sig-node", "status": "implementable", "stage": "stable", "latest_milestone": "v1.28", "milestone_alpha": "v1.21", "milestone_beta": "v1.22", "milestone_stable": "v1.28", "creation_date": "2021-01-07", "metadata_invalid": false }	2	4	{ "1": { "feature_gates": 1, "sigs": 1, "people": 2 } }	not in graph (merged pull requests only)	none	none	none	none
none	none	none	none	none	1	[ { "label": "FeatureGate", "key": "ProbeTerminationGracePeriod", "title": "ProbeTerminationGracePeriod", "url": "https://github.com/kubernetes/enhancements/blob/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period/kep.yaml", "evidence_id": "4vcqnpqixnur", "name": "ProbeTerminationGracePeriod" } ]	[ { "label": "SIG", "key": "sig-node", "title": "Node", "url": "https://github.com/kubernetes/community/tree/440e496d3a172234d81728df1bfe43849b589adf/sig-node", "evidence_id": "nrsuxu2xcxa7", "display_name": "Node", "sig_label": "node", "in_scope": true } ]	[ { "label": "Person", "key": "ehashman", "title": "ehashman", "url": "https://github.com/ehashman", "evidence_id": "twmg5pl4xllb", "login": "ehashman", "is_bot": false }, { "label": "Person", "key": "derekwaynecarr", "title": "derekwaynecarr", "url": "https://github.com/derekwaynecarr", "evidence_id": "wpnbg5qnq5fl", "login": "derekwaynecarr", "is_bot": false } ]

Rows returned by the tool
Kep	Depth	Reached	By hop	Open pull requests	Hop	Feature gates	Sigs	People
{ "label": "KEP", "key": "keps/sig-node/2238-liveness-probe-grace-period", "title": "Liveness Probe Grace Period", "url": "https://github.com/kubernetes/enhancements/tree/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period", "evidence_id": "wqcp6l675lzv", "number": 2238, "owning_sig": "sig-node", "status": "implementable", "stage": "stable", "latest_milestone": "v1.28", "milestone_alpha": "v1.21", "milestone_beta": "v1.22", "milestone_stable": "v1.28", "creation_date": "2021-01-07", "metadata_invalid": false }	2	4	{ "1": { "feature_gates": 1, "sigs": 1, "people": 2 } }	not in graph (merged pull requests only)	none	none	none	none
none	none	none	none	none	1	[ { "label": "FeatureGate", "key": "ProbeTerminationGracePeriod", "title": "ProbeTerminationGracePeriod", "url": "https://github.com/kubernetes/enhancements/blob/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period/kep.yaml", "evidence_id": "4vcqnpqixnur", "name": "ProbeTerminationGracePeriod" } ]	[ { "label": "SIG", "key": "sig-node", "title": "Node", "url": "https://github.com/kubernetes/community/tree/440e496d3a172234d81728df1bfe43849b589adf/sig-node", "evidence_id": "nrsuxu2xcxa7", "display_name": "Node", "sig_label": "node", "in_scope": true } ]	[ { "label": "Person", "key": "ehashman", "title": "ehashman", "url": "https://github.com/ehashman", "evidence_id": "twmg5pl4xllb", "login": "ehashman", "is_bot": false }, { "label": "Person", "key": "derekwaynecarr", "title": "derekwaynecarr", "url": "https://github.com/derekwaynecarr", "evidence_id": "wpnbg5qnq5fl", "login": "derekwaynecarr", "is_bot": false } ]

Kep

Depth

Reached

By hop

Open pull requests

Hop

Feature gates

Sigs

People

{
  "label": "KEP",
  "key": "keps/sig-node/2238-liveness-probe-grace-period",
  "title": "Liveness Probe Grace Period",
  "url": "https://github.com/kubernetes/enhancements/tree/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period",
  "evidence_id": "wqcp6l675lzv",
  "number": 2238,
  "owning_sig": "sig-node",
  "status": "implementable",
  "stage": "stable",
  "latest_milestone": "v1.28",
  "milestone_alpha": "v1.21",
  "milestone_beta": "v1.22",
  "milestone_stable": "v1.28",
  "creation_date": "2021-01-07",
  "metadata_invalid": false
}

{
  "1": {
    "feature_gates": 1,
    "sigs": 1,
    "people": 2
  }
}

not in graph (merged pull requests only)

[
  {
    "label": "FeatureGate",
    "key": "ProbeTerminationGracePeriod",
    "title": "ProbeTerminationGracePeriod",
    "url": "https://github.com/kubernetes/enhancements/blob/5b503a75369bdfdd494a2c2c72107a41bf6dd1a8/keps/sig-node/2238-liveness-probe-grace-period/kep.yaml",
    "evidence_id": "4vcqnpqixnur",
    "name": "ProbeTerminationGracePeriod"
  }
]

[
  {
    "label": "SIG",
    "key": "sig-node",
    "title": "Node",
    "url": "https://github.com/kubernetes/community/tree/440e496d3a172234d81728df1bfe43849b589adf/sig-node",
    "evidence_id": "nrsuxu2xcxa7",
    "display_name": "Node",
    "sig_label": "node",
    "in_scope": true
  }
]

[
  {
    "label": "Person",
    "key": "ehashman",
    "title": "ehashman",
    "url": "https://github.com/ehashman",
    "evidence_id": "twmg5pl4xllb",
    "login": "ehashman",
    "is_bot": false
  },
  {
    "label": "Person",
    "key": "derekwaynecarr",
    "title": "derekwaynecarr",
    "url": "https://github.com/derekwaynecarr",
    "evidence_id": "wpnbg5qnq5fl",
    "login": "derekwaynecarr",
    "is_bot": false
  }
]

MATCH (k:KEP {key: $key}) RETURN k LIMIT 1 ; MATCH (k:KEP {key: $key}) MATCH path = (k)-[:REPLACES|SUPERSEDED_BY|SEE_ALSO|GATED_BY*1..2]-(n) WHERE n <> k AND (n:KEP OR n:FeatureGate) AND all(x IN nodes(path)[1..-1] WHERE x <> k AND (x:KEP OR (x:FeatureGate AND NOT x.key IN $placeholders))) WITH n, path ORDER BY length(path), [x IN nodes(path) | x.key] WITH n, collect(path)[0] AS path RETURN n AS node, length(path) AS hop, path ORDER BY hop, n.key LIMIT 150 ; UNWIND $keys AS kk MATCH (k:KEP {key: kk}) RETURN k.key AS kep, [(k)-[r_0:OWNED_BY]->(s_0:SIG) WHERE coalesce(r_0.valid_to, $open) = $open | s_0] AS sigs, [(p_1:Person)-[r_1:AUTHORED]->(k) WHERE coalesce(r_1.valid_to, $open) = $open AND coalesce(p_1.is_bot, false) = false | p_1] AS authors, [(p_2:Person)-[r_2:APPROVES]->(k) WHERE coalesce(r_2.valid_to, $open) = $open AND coalesce(p_2.is_bot, false) = false | p_2] AS approvers, [(k)-[r_3:TRACKED_BY]->(i_3:Issue) WHERE coalesce(r_3.valid_to, $open) = $open AND i_3.state = 'open' | i_3] AS open_issues LIMIT 200

Liveness Probe Grace Period

Facts

How they connect

Impact of change

Keyboard shortcuts

Everywhere

Go to a page

Ask

Search lists

KEP page

Findings table

How they connect

Impact of change

Hop 1

Impact of change

Impact of change

Hop 1