Starting multiple connectors in Kafka Connect withing single distributed worker?
How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?
Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).
Currently, I am adding one connector at a time using following curl command.
curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'
Issue:
- For each new connector I add, previous/existing connector(s) restarts along with new connector.
Question:
- How should I start/create all these new connectors with one REST call in a distributed worker mode?
- Is there any way to have all connector configs in a single REST call, like an array of connector configs?
I tried to search for the same but didn't come across any workaround for this.
Thanks.
apache-kafka apache-kafka-connect
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
add a comment |
How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?
Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).
Currently, I am adding one connector at a time using following curl command.
curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'
Issue:
- For each new connector I add, previous/existing connector(s) restarts along with new connector.
Question:
- How should I start/create all these new connectors with one REST call in a distributed worker mode?
- Is there any way to have all connector configs in a single REST call, like an array of connector configs?
I tried to search for the same but didn't come across any workaround for this.
Thanks.
apache-kafka apache-kafka-connect
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
add a comment |
How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?
Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).
Currently, I am adding one connector at a time using following curl command.
curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'
Issue:
- For each new connector I add, previous/existing connector(s) restarts along with new connector.
Question:
- How should I start/create all these new connectors with one REST call in a distributed worker mode?
- Is there any way to have all connector configs in a single REST call, like an array of connector configs?
I tried to search for the same but didn't come across any workaround for this.
Thanks.
apache-kafka apache-kafka-connect
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
How to start multiple Kafka connectors in a Kafka Connect world within a single distributed worker(running on 3 different servers)?
Right now I have a need of 4 Kafka Connectors in this distributed worker(same group.id).
Currently, I am adding one connector at a time using following curl command.
curl -X POST -H "Content-type: application/json" -d '<my_single_connector_config>' 'http://localhost:8083/connectors'
Issue:
- For each new connector I add, previous/existing connector(s) restarts along with new connector.
Question:
- How should I start/create all these new connectors with one REST call in a distributed worker mode?
- Is there any way to have all connector configs in a single REST call, like an array of connector configs?
I tried to search for the same but didn't come across any workaround for this.
Thanks.
apache-kafka apache-kafka-connect
apache-kafka apache-kafka-connect
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
edited Nov 28 '18 at 20:00
cricket_007
83.6k1147116
83.6k1147116
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
asked Nov 28 '18 at 17:58
suraj_falesuraj_fale
61411743
61411743
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
For each new connector I add, previous/existing connector(s) restarts along with new connector.
Yes, that's the current behaviour of Kafka Connect. For further discussion see:
- https://issues.apache.org/jira/browse/KAFKA-5505
- https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
How should I start/create all these new connectors with one REST call in a distributed worker mode?
Is there any way to have all connector configs in a single REST call, like an array of connector configs?
You can't do it in a single REST call
If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.
So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.
Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53525448%2fstarting-multiple-connectors-in-kafka-connect-withing-single-distributed-worker%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
For each new connector I add, previous/existing connector(s) restarts along with new connector.
Yes, that's the current behaviour of Kafka Connect. For further discussion see:
- https://issues.apache.org/jira/browse/KAFKA-5505
- https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
How should I start/create all these new connectors with one REST call in a distributed worker mode?
Is there any way to have all connector configs in a single REST call, like an array of connector configs?
You can't do it in a single REST call
If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.
So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.
Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
add a comment |
For each new connector I add, previous/existing connector(s) restarts along with new connector.
Yes, that's the current behaviour of Kafka Connect. For further discussion see:
- https://issues.apache.org/jira/browse/KAFKA-5505
- https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
How should I start/create all these new connectors with one REST call in a distributed worker mode?
Is there any way to have all connector configs in a single REST call, like an array of connector configs?
You can't do it in a single REST call
If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.
So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.
Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
add a comment |
For each new connector I add, previous/existing connector(s) restarts along with new connector.
Yes, that's the current behaviour of Kafka Connect. For further discussion see:
- https://issues.apache.org/jira/browse/KAFKA-5505
- https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
How should I start/create all these new connectors with one REST call in a distributed worker mode?
Is there any way to have all connector configs in a single REST call, like an array of connector configs?
You can't do it in a single REST call
If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.
So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.
Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
For each new connector I add, previous/existing connector(s) restarts along with new connector.
Yes, that's the current behaviour of Kafka Connect. For further discussion see:
- https://issues.apache.org/jira/browse/KAFKA-5505
- https://cwiki.apache.org/confluence/display/KAFKA/Incremental+Cooperative+Rebalancing%3A+Support+and+Policies
How should I start/create all these new connectors with one REST call in a distributed worker mode?
Is there any way to have all connector configs in a single REST call, like an array of connector configs?
You can't do it in a single REST call
If you want to isolate your connectors from each other when creating/updating them, you can just run multiple distributed clusters.
So instead of 1 distributed Connect cluster running 3 connectors, you could have 3 distributed Connect clusters each running 1 connector.
Remember in practice a 'distributed Cluster' could just be of a single node, and indeed could all run on the same machine. You'd scale out for resilience and throughput capacity.
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
answered Nov 28 '18 at 18:28
Robin MoffattRobin Moffatt
8,0781429
8,0781429
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
add a comment |
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
Thanks, that's what I thought. I hope in future they will have some solution or workaround for this issue.
– suraj_fale
Nov 28 '18 at 20:26
2
2
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
If you open a JIRA on Apache Kafka (issues.apache.org/jira/projects/KAFKA/issues), there's a chance we'll improve the REST API and support requests to create multiple connectors at once. Sounds like a good idea and even easy to do. In addition, we are working avoiding all the restarts when you create a new connector.
– Gwen Shapira
Nov 28 '18 at 23:36
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
Thanks, I created JIRA as Improvement here issues.apache.org/jira/projects/KAFKA/issues/KAFKA-7835
– suraj_fale
Jan 25 at 16:55
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53525448%2fstarting-multiple-connectors-in-kafka-connect-withing-single-distributed-worker%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
