Spark Structured Streaming error Error reading field 'topic_metadata'
0
I am trying to run a very simple example. I have a Kafka readStream that reads from a Kafka topic. I am running spark 2.4.0 and Kafka 0.10.2
var streamingInputDF =
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "twitter-topic")
.load()
A console writeStream:
val activityQuery = streamingInputDF.writeStream
.format("console")
.outputMode("append")
.start()
activityQuery.awaitTermination()
However when I start the console writeStream I get the following exception
org.apache.spark.sql.streaming.StreamingQueryException: Query [id = d21cd9b4-7f51-4f5f-acbf-943dfaaeb7e5, runId = c2b2c58d-7afe-4ca5-bc36-6a3f496c19b3] terminated with exception: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
apache-spark apache-kafka spark-structured-streaming
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
add a comment |
0
I am trying to run a very simple example. I have a Kafka readStream that reads from a Kafka topic. I am running spark 2.4.0 and Kafka 0.10.2
var streamingInputDF =
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "twitter-topic")
.load()
A console writeStream:
val activityQuery = streamingInputDF.writeStream
.format("console")
.outputMode("append")
.start()
activityQuery.awaitTermination()
However when I start the console writeStream I get the following exception
org.apache.spark.sql.streaming.StreamingQueryException: Query [id = d21cd9b4-7f51-4f5f-acbf-943dfaaeb7e5, runId = c2b2c58d-7afe-4ca5-bc36-6a3f496c19b3] terminated with exception: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
apache-spark apache-kafka spark-structured-streaming
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59
add a comment |
0
0
0
I am trying to run a very simple example. I have a Kafka readStream that reads from a Kafka topic. I am running spark 2.4.0 and Kafka 0.10.2
var streamingInputDF =
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "twitter-topic")
.load()
A console writeStream:
val activityQuery = streamingInputDF.writeStream
.format("console")
.outputMode("append")
.start()
activityQuery.awaitTermination()
However when I start the console writeStream I get the following exception
org.apache.spark.sql.streaming.StreamingQueryException: Query [id = d21cd9b4-7f51-4f5f-acbf-943dfaaeb7e5, runId = c2b2c58d-7afe-4ca5-bc36-6a3f496c19b3] terminated with exception: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
apache-spark apache-kafka spark-structured-streaming
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
I am trying to run a very simple example. I have a Kafka readStream that reads from a Kafka topic. I am running spark 2.4.0 and Kafka 0.10.2
var streamingInputDF =
spark.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "localhost:9092")
.option("subscribe", "twitter-topic")
.load()
A console writeStream:
val activityQuery = streamingInputDF.writeStream
.format("console")
.outputMode("append")
.start()
activityQuery.awaitTermination()
However when I start the console writeStream I get the following exception
org.apache.spark.sql.streaming.StreamingQueryException: Query [id = d21cd9b4-7f51-4f5f-acbf-943dfaaeb7e5, runId = c2b2c58d-7afe-4ca5-bc36-6a3f496c19b3] terminated with exception: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.spark.sql.execution.streaming.StreamExecution.org$apache$spark$sql$execution$streaming$StreamExecution$$runStream(StreamExecution.scala:295)
at org.apache.spark.sql.execution.streaming.StreamExecution$$anon$1.run(StreamExecution.scala:189)
Caused by: org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'topic_metadata': Error reading array of size 881783, only 41 bytes available
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
apache-spark apache-kafka spark-structured-streaming
apache-spark apache-kafka spark-structured-streaming
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
edited Nov 25 '18 at 8:15
user6910411
33.2k97398
33.2k97398
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
asked Nov 24 '18 at 22:50
danoomistmatistedanoomistmatiste
214
214
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59
add a comment |
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59
add a comment |
1 Answer
1
active
oldest
votes
0
I added kafka-clients-0.10.2.2.jar to the spark-submit command line and this error went away
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53463037%2fspark-structured-streaming-error-error-reading-field-topic-metadata%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
I added kafka-clients-0.10.2.2.jar to the spark-submit command line and this error went away
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
add a comment |
0
I added kafka-clients-0.10.2.2.jar to the spark-submit command line and this error went away
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
add a comment |
0
0
0
I added kafka-clients-0.10.2.2.jar to the spark-submit command line and this error went away
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
I added kafka-clients-0.10.2.2.jar to the spark-submit command line and this error went away
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
answered Nov 25 '18 at 5:51
danoomistmatistedanoomistmatiste
214
214
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
add a comment |
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
Spark 2.4 added Kafka 2.0 clients, by the way
– cricket_007
Nov 25 '18 at 16:30
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53463037%2fspark-structured-streaming-error-error-reading-field-topic-metadata%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Which Kafka library are you using?
– cricket_007
Nov 25 '18 at 0:57
This error usually occurs when there is some version mismatch. Please verify that all the libraries you are using are of the same version.
– anuj saxena
Nov 25 '18 at 3:59