Is it OK to make all pods in a Kubernetes statefulset fail ReadinessProbes instead of just one?
We have a statefulset for a service (Druid historicals) that caches a lot of data on local SSDs. (We run one pod per node in SSD using taints and affinity.) When we need to replace the underlying machines, this means pods start up with empty local disks and they then take a while to refill their caches. We ideally only want to do planned replacement of nodes (eg, GKE Node pool upgrade) one node at a time and wait until the pod on the new node has fully filled up its cache before rolling out the next node.
OK, so this means that we need to set a PodDisruptionBudget of 1, and set up Readiness probe to make the new node not ready until the cache has been filled.
The problem is: the system doesn't really offer a great way for us to ask the question "has pod X downloaded all the stuff it needs to have to get the system as a whole fully replicated".
What it does let us ask is "is the entire system fully replicated?".
So we are tempted to write a Readiness probe that says "not ready unless entire system is fully replicated". But this means that during node pool upgrades (or other short occasions that have brief "not fully replicated" states), every pod in the statefulset would become unready.
My question is: I don't really understand the full implications of every part of k8s that consults the Ready status. Would it be bad if every pod in the SS became unready while a single pod is "loading up"?
My understanding is that readiness is used for things like controlling the tempo of a Deployment or StatefulSet rollout (which is fine here), and that it's also used for having Services determine which pods to route to. In this case we don't actually use the Service associated with the StatefulSet for routing (clients connect directly to individual pods). So it seems like this might actually be fine. But is it? Or are there other applications of the Ready state which would make it bad for us to mark all pods as unready while global replication isn't at 100%?
kubernetes statefulset kubernetes-statefulset asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
add a comment |
We have a statefulset for a service (Druid historicals) that caches a lot of data on local SSDs. (We run one pod per node in SSD using taints and affinity.) When we need to replace the underlying machines, this means pods start up with empty local disks and they then take a while to refill their caches. We ideally only want to do planned replacement of nodes (eg, GKE Node pool upgrade) one node at a time and wait until the pod on the new node has fully filled up its cache before rolling out the next node.
OK, so this means that we need to set a PodDisruptionBudget of 1, and set up Readiness probe to make the new node not ready until the cache has been filled.
The problem is: the system doesn't really offer a great way for us to ask the question "has pod X downloaded all the stuff it needs to have to get the system as a whole fully replicated".
What it does let us ask is "is the entire system fully replicated?".
So we are tempted to write a Readiness probe that says "not ready unless entire system is fully replicated". But this means that during node pool upgrades (or other short occasions that have brief "not fully replicated" states), every pod in the statefulset would become unready.
My question is: I don't really understand the full implications of every part of k8s that consults the Ready status. Would it be bad if every pod in the SS became unready while a single pod is "loading up"?
My understanding is that readiness is used for things like controlling the tempo of a Deployment or StatefulSet rollout (which is fine here), and that it's also used for having Services determine which pods to route to. In this case we don't actually use the Service associated with the StatefulSet for routing (clients connect directly to individual pods). So it seems like this might actually be fine. But is it? Or are there other applications of the Ready state which would make it bad for us to mark all pods as unready while global replication isn't at 100%?
kubernetes statefulset kubernetes-statefulset asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same likeinitContainers, you could use it to test if replication went through successfully.
– Crou
Nov 29 '18 at 14:31
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until thenPODshould in not ready. But those are the things you have to do.
– Crou
Dec 5 '18 at 16:21
add a comment |
We have a statefulset for a service (Druid historicals) that caches a lot of data on local SSDs. (We run one pod per node in SSD using taints and affinity.) When we need to replace the underlying machines, this means pods start up with empty local disks and they then take a while to refill their caches. We ideally only want to do planned replacement of nodes (eg, GKE Node pool upgrade) one node at a time and wait until the pod on the new node has fully filled up its cache before rolling out the next node.
OK, so this means that we need to set a PodDisruptionBudget of 1, and set up Readiness probe to make the new node not ready until the cache has been filled.
The problem is: the system doesn't really offer a great way for us to ask the question "has pod X downloaded all the stuff it needs to have to get the system as a whole fully replicated".
What it does let us ask is "is the entire system fully replicated?".
So we are tempted to write a Readiness probe that says "not ready unless entire system is fully replicated". But this means that during node pool upgrades (or other short occasions that have brief "not fully replicated" states), every pod in the statefulset would become unready.
My question is: I don't really understand the full implications of every part of k8s that consults the Ready status. Would it be bad if every pod in the SS became unready while a single pod is "loading up"?
My understanding is that readiness is used for things like controlling the tempo of a Deployment or StatefulSet rollout (which is fine here), and that it's also used for having Services determine which pods to route to. In this case we don't actually use the Service associated with the StatefulSet for routing (clients connect directly to individual pods). So it seems like this might actually be fine. But is it? Or are there other applications of the Ready state which would make it bad for us to mark all pods as unready while global replication isn't at 100%?
kubernetes statefulset kubernetes-statefulset asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
We have a statefulset for a service (Druid historicals) that caches a lot of data on local SSDs. (We run one pod per node in SSD using taints and affinity.) When we need to replace the underlying machines, this means pods start up with empty local disks and they then take a while to refill their caches. We ideally only want to do planned replacement of nodes (eg, GKE Node pool upgrade) one node at a time and wait until the pod on the new node has fully filled up its cache before rolling out the next node.
OK, so this means that we need to set a PodDisruptionBudget of 1, and set up Readiness probe to make the new node not ready until the cache has been filled.
The problem is: the system doesn't really offer a great way for us to ask the question "has pod X downloaded all the stuff it needs to have to get the system as a whole fully replicated".
What it does let us ask is "is the entire system fully replicated?".
So we are tempted to write a Readiness probe that says "not ready unless entire system is fully replicated". But this means that during node pool upgrades (or other short occasions that have brief "not fully replicated" states), every pod in the statefulset would become unready.
My question is: I don't really understand the full implications of every part of k8s that consults the Ready status. Would it be bad if every pod in the SS became unready while a single pod is "loading up"?
My understanding is that readiness is used for things like controlling the tempo of a Deployment or StatefulSet rollout (which is fine here), and that it's also used for having Services determine which pods to route to. In this case we don't actually use the Service associated with the StatefulSet for routing (clients connect directly to individual pods). So it seems like this might actually be fine. But is it? Or are there other applications of the Ready state which would make it bad for us to mark all pods as unready while global replication isn't at 100%?
kubernetes statefulset kubernetes-statefulset kubernetes statefulset kubernetes-statefulset asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
asked Nov 28 '18 at 18:42
David GlasserDavid Glasser
1,3391119
1,3391119
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same likeinitContainers, you could use it to test if replication went through successfully.
– Crou
Nov 29 '18 at 14:31
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until thenPODshould in not ready. But those are the things you have to do.
– Crou
Dec 5 '18 at 16:21
add a comment |
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same likeinitContainers, you could use it to test if replication went through successfully.
– Crou
Nov 29 '18 at 14:31
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until thenPODshould in not ready. But those are the things you have to do.
– Crou
Dec 5 '18 at 16:21
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same like
initContainers, you could use it to test if replication went through successfully.– Crou
Nov 29 '18 at 14:31
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same like
initContainers, you could use it to test if replication went through successfully.– Crou
Nov 29 '18 at 14:31
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until then
POD should in not ready. But those are the things you have to do.– Crou
Dec 5 '18 at 16:21
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until then
POD should in not ready. But those are the things you have to do.– Crou
Dec 5 '18 at 16:21
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53526075%2fis-it-ok-to-make-all-pods-in-a-kubernetes-statefulset-fail-readinessprobes-inste%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53526075%2fis-it-ok-to-make-all-pods-in-a-kubernetes-statefulset-fail-readinessprobes-inste%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
I'm not sure why would you make all pods fail ReadinessProbe, it can be used to setup dependencies. Same like
initContainers, you could use it to test if replication went through successfully.– Crou
Nov 29 '18 at 14:31
The underlying software doesn't make it easy to ask the question "does this pod have all the data it should have". It makes it easy to ask the question "is the cluster as a whole under-replicated". But I don't think it really has a concept of the individual node's requirements, just "we need two copies of every file somewhere".
– David Glasser
Nov 29 '18 at 19:08
You should create your own image, with a script running inside that will output '0' if cache is correctly replicated. Kubernetes will be spin the image and wait for this output, until then
PODshould in not ready. But those are the things you have to do.– Crou
Dec 5 '18 at 16:21