mongodb collection is not loaded into hive table
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ height:90px;width:728px;box-sizing:border-box;
}
0
I am currently trying to migrate mongodb collection into hdfs by using hive.
I succeeded in creating the mongodb-based hive table and it is present in my hdfs directory when i browse the hdfs UI.
However only the table is showing but the table is empty, no data is available.
It seems like the mongodb collection is not loaded into the table. below is how I create the table.
create table mongo(
id String,
age String,
gender String,
race String,
custState String,
purchaseDate String,
purchaseTime String,
foodname String,
restaurant String,
foodtype String,
quantity String,
totalPrice String,
orderType String,
rating String,
servingType String,
characteristic String,
restaurantType String,
restaurantState String,
priceRange String,
paymentMethod String,
tableBooking String,
onlineBooking String,
deliveryService String)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{
"id":"_id",
"age":"age",
"gender":"gender",
"race":"race",
"custState":"custState ",
"purchaseDate":"purchaseDate",
"purchaseTime":"purchaseTime",
"foodname":"foodname",
"restaurant":"restaurant",
"foodtype":"foodtype",
"quantity":"quantity",
"totalPrice":"totalPrice",
"orderType":"orderType",
"rating":"rating",
"servingType":"servingType",
"characteristic":"characteristic",
"restaurantType":"restaurantType",
"restaurantState":"restaurantState",
"priceRange":"priceRange",
"paymentMethod":"paymentMethod",
"tableBooking":"tableBooking",
"onlineBooking":"onlineBooking",
"deliveryService":"deliveryService"}')
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');
before creating the table I also added jars as suggested tho I am not properly sure what's the significance of it.
add jar /usr/local/hadoop/lib/hive-serde-1.0.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-core-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-hive-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-java-driver-3.8.1.jar;
add jar /usr/local/Hive-JSON-Serde/json-serde/target/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar;
I am using hadoop 2.9.1, mongodb 4.0.4, and hive 2.3.3. Can anybody help me to fix this issue?
mongodb hadoop hive
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
add a comment |
0
I am currently trying to migrate mongodb collection into hdfs by using hive.
I succeeded in creating the mongodb-based hive table and it is present in my hdfs directory when i browse the hdfs UI.
However only the table is showing but the table is empty, no data is available.
It seems like the mongodb collection is not loaded into the table. below is how I create the table.
create table mongo(
id String,
age String,
gender String,
race String,
custState String,
purchaseDate String,
purchaseTime String,
foodname String,
restaurant String,
foodtype String,
quantity String,
totalPrice String,
orderType String,
rating String,
servingType String,
characteristic String,
restaurantType String,
restaurantState String,
priceRange String,
paymentMethod String,
tableBooking String,
onlineBooking String,
deliveryService String)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{
"id":"_id",
"age":"age",
"gender":"gender",
"race":"race",
"custState":"custState ",
"purchaseDate":"purchaseDate",
"purchaseTime":"purchaseTime",
"foodname":"foodname",
"restaurant":"restaurant",
"foodtype":"foodtype",
"quantity":"quantity",
"totalPrice":"totalPrice",
"orderType":"orderType",
"rating":"rating",
"servingType":"servingType",
"characteristic":"characteristic",
"restaurantType":"restaurantType",
"restaurantState":"restaurantState",
"priceRange":"priceRange",
"paymentMethod":"paymentMethod",
"tableBooking":"tableBooking",
"onlineBooking":"onlineBooking",
"deliveryService":"deliveryService"}')
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');
before creating the table I also added jars as suggested tho I am not properly sure what's the significance of it.
add jar /usr/local/hadoop/lib/hive-serde-1.0.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-core-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-hive-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-java-driver-3.8.1.jar;
add jar /usr/local/Hive-JSON-Serde/json-serde/target/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar;
I am using hadoop 2.9.1, mongodb 4.0.4, and hive 2.3.3. Can anybody help me to fix this issue?
mongodb hadoop hive
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21
add a comment |
0
0
0
I am currently trying to migrate mongodb collection into hdfs by using hive.
I succeeded in creating the mongodb-based hive table and it is present in my hdfs directory when i browse the hdfs UI.
However only the table is showing but the table is empty, no data is available.
It seems like the mongodb collection is not loaded into the table. below is how I create the table.
create table mongo(
id String,
age String,
gender String,
race String,
custState String,
purchaseDate String,
purchaseTime String,
foodname String,
restaurant String,
foodtype String,
quantity String,
totalPrice String,
orderType String,
rating String,
servingType String,
characteristic String,
restaurantType String,
restaurantState String,
priceRange String,
paymentMethod String,
tableBooking String,
onlineBooking String,
deliveryService String)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{
"id":"_id",
"age":"age",
"gender":"gender",
"race":"race",
"custState":"custState ",
"purchaseDate":"purchaseDate",
"purchaseTime":"purchaseTime",
"foodname":"foodname",
"restaurant":"restaurant",
"foodtype":"foodtype",
"quantity":"quantity",
"totalPrice":"totalPrice",
"orderType":"orderType",
"rating":"rating",
"servingType":"servingType",
"characteristic":"characteristic",
"restaurantType":"restaurantType",
"restaurantState":"restaurantState",
"priceRange":"priceRange",
"paymentMethod":"paymentMethod",
"tableBooking":"tableBooking",
"onlineBooking":"onlineBooking",
"deliveryService":"deliveryService"}')
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');
before creating the table I also added jars as suggested tho I am not properly sure what's the significance of it.
add jar /usr/local/hadoop/lib/hive-serde-1.0.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-core-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-hive-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-java-driver-3.8.1.jar;
add jar /usr/local/Hive-JSON-Serde/json-serde/target/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar;
I am using hadoop 2.9.1, mongodb 4.0.4, and hive 2.3.3. Can anybody help me to fix this issue?
mongodb hadoop hive
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
I am currently trying to migrate mongodb collection into hdfs by using hive.
I succeeded in creating the mongodb-based hive table and it is present in my hdfs directory when i browse the hdfs UI.
However only the table is showing but the table is empty, no data is available.
It seems like the mongodb collection is not loaded into the table. below is how I create the table.
create table mongo(
id String,
age String,
gender String,
race String,
custState String,
purchaseDate String,
purchaseTime String,
foodname String,
restaurant String,
foodtype String,
quantity String,
totalPrice String,
orderType String,
rating String,
servingType String,
characteristic String,
restaurantType String,
restaurantState String,
priceRange String,
paymentMethod String,
tableBooking String,
onlineBooking String,
deliveryService String)
ROW FORMAT SERDE 'org.openx.data.jsonserde.JsonSerDe'
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{
"id":"_id",
"age":"age",
"gender":"gender",
"race":"race",
"custState":"custState ",
"purchaseDate":"purchaseDate",
"purchaseTime":"purchaseTime",
"foodname":"foodname",
"restaurant":"restaurant",
"foodtype":"foodtype",
"quantity":"quantity",
"totalPrice":"totalPrice",
"orderType":"orderType",
"rating":"rating",
"servingType":"servingType",
"characteristic":"characteristic",
"restaurantType":"restaurantType",
"restaurantState":"restaurantState",
"priceRange":"priceRange",
"paymentMethod":"paymentMethod",
"tableBooking":"tableBooking",
"onlineBooking":"onlineBooking",
"deliveryService":"deliveryService"}')
TBLPROPERTIES('mongo.uri'='mongodb://master:27017/bigdatadb.datasetv1');
before creating the table I also added jars as suggested tho I am not properly sure what's the significance of it.
add jar /usr/local/hadoop/lib/hive-serde-1.0.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-core-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-hadoop-hive-1.5.2.jar;
add jar /usr/local/hadoop/lib/mongo-java-driver-3.8.1.jar;
add jar /usr/local/Hive-JSON-Serde/json-serde/target/json-serde-1.3.9-SNAPSHOT-jar-with-dependencies.jar;
I am using hadoop 2.9.1, mongodb 4.0.4, and hive 2.3.3. Can anybody help me to fix this issue?
mongodb hadoop hive
mongodb hadoop hive
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
edited Nov 29 '18 at 3:01
DINA TAKLIT
743823
743823
asked Nov 29 '18 at 2:51
gdphygdphy
2418
asked Nov 29 '18 at 2:51
gdphygdphy
2418
asked Nov 29 '18 at 2:51
gdphygdphy
2418
2418
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21
add a comment |
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21
add a comment |
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53531142%2fmongodb-collection-is-not-loaded-into-hive-table%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53531142%2fmongodb-collection-is-not-loaded-into-hive-table%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
just to confirm, you are not getting any error, do you ?
– Gaurang Shah
Nov 29 '18 at 4:12
Yes, didn't get any error
– gdphy
Nov 29 '18 at 4:21