“errors.deadletterqueue.topic.name” no effect in Confluent 5.0.1
I have upgraded from confluent 4.0.1 to confluent 5.0.1 recently. The bootstrap.server's version is Kafka 1.0.
In my HBaseSink Connector, I have configured the new feature ""errors.deadletterqueue.topic.name" as follows:
{
"name": "kafka-hbase4",
"config": {
"connector.class": "com.xxxx.hbase.sink.HBaseSinkConnector",
....
"zookeeper.quorum": "xxxx:2181",
"topics": "oplog_demo.hbase_test1",
"errors.deadletterqueue.topic.name":"error-topic-1"
}
}
the program normally works O.K. However, when I send some error messages to the topic of the HBase Sink Connector, like some Chineses words "哈哈哈", the program goes down, throwing a SerializationException, and no messages are consumed by the "error-topic-1".
Does that mean the configuration "errors.deadletterqueue.topic.name" not working? How come could that be and how can I make this configuration work?
apache-kafka apache-kafka-connect confluent
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
add a comment |
I have upgraded from confluent 4.0.1 to confluent 5.0.1 recently. The bootstrap.server's version is Kafka 1.0.
In my HBaseSink Connector, I have configured the new feature ""errors.deadletterqueue.topic.name" as follows:
{
"name": "kafka-hbase4",
"config": {
"connector.class": "com.xxxx.hbase.sink.HBaseSinkConnector",
....
"zookeeper.quorum": "xxxx:2181",
"topics": "oplog_demo.hbase_test1",
"errors.deadletterqueue.topic.name":"error-topic-1"
}
}
the program normally works O.K. However, when I send some error messages to the topic of the HBase Sink Connector, like some Chineses words "哈哈哈", the program goes down, throwing a SerializationException, and no messages are consumed by the "error-topic-1".
Does that mean the configuration "errors.deadletterqueue.topic.name" not working? How come could that be and how can I make this configuration work?
apache-kafka apache-kafka-connect confluent
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
1
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33
add a comment |
I have upgraded from confluent 4.0.1 to confluent 5.0.1 recently. The bootstrap.server's version is Kafka 1.0.
In my HBaseSink Connector, I have configured the new feature ""errors.deadletterqueue.topic.name" as follows:
{
"name": "kafka-hbase4",
"config": {
"connector.class": "com.xxxx.hbase.sink.HBaseSinkConnector",
....
"zookeeper.quorum": "xxxx:2181",
"topics": "oplog_demo.hbase_test1",
"errors.deadletterqueue.topic.name":"error-topic-1"
}
}
the program normally works O.K. However, when I send some error messages to the topic of the HBase Sink Connector, like some Chineses words "哈哈哈", the program goes down, throwing a SerializationException, and no messages are consumed by the "error-topic-1".
Does that mean the configuration "errors.deadletterqueue.topic.name" not working? How come could that be and how can I make this configuration work?
apache-kafka apache-kafka-connect confluent
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
I have upgraded from confluent 4.0.1 to confluent 5.0.1 recently. The bootstrap.server's version is Kafka 1.0.
In my HBaseSink Connector, I have configured the new feature ""errors.deadletterqueue.topic.name" as follows:
{
"name": "kafka-hbase4",
"config": {
"connector.class": "com.xxxx.hbase.sink.HBaseSinkConnector",
....
"zookeeper.quorum": "xxxx:2181",
"topics": "oplog_demo.hbase_test1",
"errors.deadletterqueue.topic.name":"error-topic-1"
}
}
the program normally works O.K. However, when I send some error messages to the topic of the HBase Sink Connector, like some Chineses words "哈哈哈", the program goes down, throwing a SerializationException, and no messages are consumed by the "error-topic-1".
Does that mean the configuration "errors.deadletterqueue.topic.name" not working? How come could that be and how can I make this configuration work?
apache-kafka apache-kafka-connect confluent
apache-kafka apache-kafka-connect confluent
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
edited Nov 26 '18 at 7:09
cricket_007
81.5k1142111
81.5k1142111
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
asked Nov 26 '18 at 6:32
SkyOneSkyOne
85
85
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
1
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33
add a comment |
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
1
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
1
1
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53475797%2ferrors-deadletterqueue-topic-name-no-effect-in-confluent-5-0-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53475797%2ferrors-deadletterqueue-topic-name-no-effect-in-confluent-5-0-1%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Not clear what serializers you are using, but the StringSerializer expects UTF-8 data, no matter what topic you've configured
– cricket_007
Nov 26 '18 at 7:10
@cricket_007 I am using the default "org.apache.kafka.connect.json.JsonConverter". Actually, I'm expecting a serialization exception and the message can be sent to the deadletterqueue topic, but what baffles me is nothing happens.
– SkyOne
Nov 26 '18 at 7:32
I haven't used the DLQ feature, but I think it still tries to apply a serializer to those messages to send to the other topic rather than just copy the raw bytes
– cricket_007
Nov 26 '18 at 18:03
1
this problem is due to my java classpath set wrong. After I set the right path, problem solved.
– SkyOne
Nov 27 '18 at 12:23
Cool. Feel free to provide the full solution below as an answer rather than a comment
– cricket_007
Nov 27 '18 at 17:33