Save the prediction output of Tensorflow model into hdfs file
I have a trained a tf model and I want to apply it to big dataset in hdfs which is about billion of samples. The main point is I need to write the prediction of tf model into hdfs file. However I can't find the relative API in tensorflow about how to save data in hdfs file, only find the api about reading hdfs file
Until now the way I did it is to save the trained tf model into pb file in local and then load the pb file using Java api in spark or Mapreduce code. The problem of both spark or mapreduce is the running speed is very slow and failed with exceeds memory error.
Here is my demo:
public class TF_model implements Serializable{
public Session session;
public TF_model(String model_path){
try{
Graph graph = new Graph();
InputStream stream = this.getClass().getClassLoader().getResourceAsStream(model_path);
byte graphBytes = IOUtils.toByteArray(stream);
graph.importGraphDef(graphBytes);
this.session = new Session(graph);
}
catch (Exception e){
System.out.println("failed to load tensorflow model");
}
}
// this is the function to predict a sample in hdfs
public int predict(int token_id_array){
Tensor z = session.runner()
.feed("words_ids_placeholder", Tensor.create(new int{token_id_array}))
.fetch("softmax_prediction").run().get(0);
double softmax_prediction = new double[1][token_id_array.length][2];
z.copyTo(softmax_prediction);
return softmax_prediction[0];
}}
below is my spark code:
val rdd = spark.sparkContext.textFile(file_path)
val predct_result= rdd.mapPartitions(pa=>{
val tf_model = new TF_model("model.pb")
pa.map(line=>{
val transformed = transform(line) // omitted the transform code
val rs = tf_model .predict(transformed)
rs
})
})
I also tried tensorflow deployed in hadoop, but can't find a way to write big dataset into HDFS.
apache-spark hadoop tensorflow hdfs distribution
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
add a comment |
I have a trained a tf model and I want to apply it to big dataset in hdfs which is about billion of samples. The main point is I need to write the prediction of tf model into hdfs file. However I can't find the relative API in tensorflow about how to save data in hdfs file, only find the api about reading hdfs file
Until now the way I did it is to save the trained tf model into pb file in local and then load the pb file using Java api in spark or Mapreduce code. The problem of both spark or mapreduce is the running speed is very slow and failed with exceeds memory error.
Here is my demo:
public class TF_model implements Serializable{
public Session session;
public TF_model(String model_path){
try{
Graph graph = new Graph();
InputStream stream = this.getClass().getClassLoader().getResourceAsStream(model_path);
byte graphBytes = IOUtils.toByteArray(stream);
graph.importGraphDef(graphBytes);
this.session = new Session(graph);
}
catch (Exception e){
System.out.println("failed to load tensorflow model");
}
}
// this is the function to predict a sample in hdfs
public int predict(int token_id_array){
Tensor z = session.runner()
.feed("words_ids_placeholder", Tensor.create(new int{token_id_array}))
.fetch("softmax_prediction").run().get(0);
double softmax_prediction = new double[1][token_id_array.length][2];
z.copyTo(softmax_prediction);
return softmax_prediction[0];
}}
below is my spark code:
val rdd = spark.sparkContext.textFile(file_path)
val predct_result= rdd.mapPartitions(pa=>{
val tf_model = new TF_model("model.pb")
pa.map(line=>{
val transformed = transform(line) // omitted the transform code
val rs = tf_model .predict(transformed)
rs
})
})
I also tried tensorflow deployed in hadoop, but can't find a way to write big dataset into HDFS.
apache-spark hadoop tensorflow hdfs distribution
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
add a comment |
I have a trained a tf model and I want to apply it to big dataset in hdfs which is about billion of samples. The main point is I need to write the prediction of tf model into hdfs file. However I can't find the relative API in tensorflow about how to save data in hdfs file, only find the api about reading hdfs file
Until now the way I did it is to save the trained tf model into pb file in local and then load the pb file using Java api in spark or Mapreduce code. The problem of both spark or mapreduce is the running speed is very slow and failed with exceeds memory error.
Here is my demo:
public class TF_model implements Serializable{
public Session session;
public TF_model(String model_path){
try{
Graph graph = new Graph();
InputStream stream = this.getClass().getClassLoader().getResourceAsStream(model_path);
byte graphBytes = IOUtils.toByteArray(stream);
graph.importGraphDef(graphBytes);
this.session = new Session(graph);
}
catch (Exception e){
System.out.println("failed to load tensorflow model");
}
}
// this is the function to predict a sample in hdfs
public int predict(int token_id_array){
Tensor z = session.runner()
.feed("words_ids_placeholder", Tensor.create(new int{token_id_array}))
.fetch("softmax_prediction").run().get(0);
double softmax_prediction = new double[1][token_id_array.length][2];
z.copyTo(softmax_prediction);
return softmax_prediction[0];
}}
below is my spark code:
val rdd = spark.sparkContext.textFile(file_path)
val predct_result= rdd.mapPartitions(pa=>{
val tf_model = new TF_model("model.pb")
pa.map(line=>{
val transformed = transform(line) // omitted the transform code
val rs = tf_model .predict(transformed)
rs
})
})
I also tried tensorflow deployed in hadoop, but can't find a way to write big dataset into HDFS.
apache-spark hadoop tensorflow hdfs distribution
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
I have a trained a tf model and I want to apply it to big dataset in hdfs which is about billion of samples. The main point is I need to write the prediction of tf model into hdfs file. However I can't find the relative API in tensorflow about how to save data in hdfs file, only find the api about reading hdfs file
Until now the way I did it is to save the trained tf model into pb file in local and then load the pb file using Java api in spark or Mapreduce code. The problem of both spark or mapreduce is the running speed is very slow and failed with exceeds memory error.
Here is my demo:
public class TF_model implements Serializable{
public Session session;
public TF_model(String model_path){
try{
Graph graph = new Graph();
InputStream stream = this.getClass().getClassLoader().getResourceAsStream(model_path);
byte graphBytes = IOUtils.toByteArray(stream);
graph.importGraphDef(graphBytes);
this.session = new Session(graph);
}
catch (Exception e){
System.out.println("failed to load tensorflow model");
}
}
// this is the function to predict a sample in hdfs
public int predict(int token_id_array){
Tensor z = session.runner()
.feed("words_ids_placeholder", Tensor.create(new int{token_id_array}))
.fetch("softmax_prediction").run().get(0);
double softmax_prediction = new double[1][token_id_array.length][2];
z.copyTo(softmax_prediction);
return softmax_prediction[0];
}}
below is my spark code:
val rdd = spark.sparkContext.textFile(file_path)
val predct_result= rdd.mapPartitions(pa=>{
val tf_model = new TF_model("model.pb")
pa.map(line=>{
val transformed = transform(line) // omitted the transform code
val rs = tf_model .predict(transformed)
rs
})
})
I also tried tensorflow deployed in hadoop, but can't find a way to write big dataset into HDFS.
apache-spark hadoop tensorflow hdfs distribution
apache-spark hadoop tensorflow hdfs distribution
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
edited Dec 8 '18 at 2:29
Richard. Zhu
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
asked Nov 25 '18 at 13:13
Richard. ZhuRichard. Zhu
65
65
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
You may read model file from hdfs one time, then use sc.broadcast your bytes array of your graph to partitions. Finally, start load graph and predict. Just to avoid read file multiple time from hdfs.
answered Dec 17 '18 at 3:20
LirenLiren
1
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467817%2fsave-the-prediction-output-of-tensorflow-model-into-hdfs-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
You may read model file from hdfs one time, then use sc.broadcast your bytes array of your graph to partitions. Finally, start load graph and predict. Just to avoid read file multiple time from hdfs.
answered Dec 17 '18 at 3:20
LirenLiren
1
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
add a comment |
You may read model file from hdfs one time, then use sc.broadcast your bytes array of your graph to partitions. Finally, start load graph and predict. Just to avoid read file multiple time from hdfs.
answered Dec 17 '18 at 3:20
LirenLiren
1
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
add a comment |
You may read model file from hdfs one time, then use sc.broadcast your bytes array of your graph to partitions. Finally, start load graph and predict. Just to avoid read file multiple time from hdfs.
answered Dec 17 '18 at 3:20
LirenLiren
1
You may read model file from hdfs one time, then use sc.broadcast your bytes array of your graph to partitions. Finally, start load graph and predict. Just to avoid read file multiple time from hdfs.
answered Dec 17 '18 at 3:20
LirenLiren
1
answered Dec 17 '18 at 3:20
LirenLiren
1
answered Dec 17 '18 at 3:20
LirenLiren
1
answered Dec 17 '18 at 3:20
LirenLiren
1
1
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
add a comment |
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
oh, sorry, you load from resources, I looked it wrong. Anyway, thanks for your IOUtils class to help me with bytes stream load problem.
– Liren
Dec 17 '18 at 3:24
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
I used Tensor.close() to fix memory leak problem, and have slow problem too.
– Liren
Dec 17 '18 at 8:51
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53467817%2fsave-the-prediction-output-of-tensorflow-model-into-hdfs-file%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
