How to calculate the difference in time between 2 row in spark Scala
-1
I am new to Spark-Scala technologies, as part of my learning, I am trying to find the time between two rows of same colum which consist of date and time together as seen below,
column1
1/1/2017 12:01:00 AM
1/1/2017 12:05:00 AM
So I want to get the time variation between two rows from row 1 and row 2 of column1 as both belongs to the same date.
Please let me know what would be the best method to achieve it?
Appreciate if anyone can help on this
Thanks
scala apache-spark
asked Nov 28 '18 at 13:27
umeshumesh
24
add a comment |
-1
I am new to Spark-Scala technologies, as part of my learning, I am trying to find the time between two rows of same colum which consist of date and time together as seen below,
column1
1/1/2017 12:01:00 AM
1/1/2017 12:05:00 AM
So I want to get the time variation between two rows from row 1 and row 2 of column1 as both belongs to the same date.
Please let me know what would be the best method to achieve it?
Appreciate if anyone can help on this
Thanks
scala apache-spark
asked Nov 28 '18 at 13:27
umeshumesh
24
1
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40
add a comment |
-1
-1
-1
I am new to Spark-Scala technologies, as part of my learning, I am trying to find the time between two rows of same colum which consist of date and time together as seen below,
column1
1/1/2017 12:01:00 AM
1/1/2017 12:05:00 AM
So I want to get the time variation between two rows from row 1 and row 2 of column1 as both belongs to the same date.
Please let me know what would be the best method to achieve it?
Appreciate if anyone can help on this
Thanks
scala apache-spark
asked Nov 28 '18 at 13:27
umeshumesh
24
I am new to Spark-Scala technologies, as part of my learning, I am trying to find the time between two rows of same colum which consist of date and time together as seen below,
column1
1/1/2017 12:01:00 AM
1/1/2017 12:05:00 AM
So I want to get the time variation between two rows from row 1 and row 2 of column1 as both belongs to the same date.
Please let me know what would be the best method to achieve it?
Appreciate if anyone can help on this
Thanks
scala apache-spark
scala apache-spark
asked Nov 28 '18 at 13:27
umeshumesh
24
asked Nov 28 '18 at 13:27
umeshumesh
24
edited Nov 28 '18 at 14:21
umesh
asked Nov 28 '18 at 13:27
umeshumesh
24
asked Nov 28 '18 at 13:27
umeshumesh
24
asked Nov 28 '18 at 13:27
umeshumesh
24
24
1
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40
add a comment |
1
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40
1
1
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40
add a comment |
1 Answer
1
active
oldest
votes
0
You need to cast the column to timestamp and then do the diff calculation.
Check this out:
scala> val df = Seq(("1/01/2017 12:01:00 AM","1/1/2017 12:05:00 AM")).toDF("time1","time2")
df: org.apache.spark.sql.DataFrame = [time1: string, time2: string]
scala> val df2 = df.withColumn("time1",to_timestamp('time1,"d/MM/yyyy hh:mm:ss a")).withColumn("time2",to_timestamp('time2,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df2.printSchema
root
|-- time1: timestamp (nullable = true)
|-- time2: timestamp (nullable = true)
scala> df2.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
scala>
Update1:
scala> val df = Seq(("1/01/2017 12:01:00 AM"),("1/1/2017 12:05:00 AM")).toDF("timex")
df: org.apache.spark.sql.DataFrame = [timex: string]
scala> val df2 = df.withColumn("timex",to_timestamp('timex,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [timex: timestamp]
scala> df2.show
+-------------------+
| timex|
+-------------------+
|2017-01-01 00:01:00|
|2017-01-01 00:05:00|
+-------------------+
scala> val df3 = df2.alias("t1").join(df2.alias("t2"), $"t1.timex" =!= $"t2.timex", "leftOuter").toDF("time1","time2")
df3: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
|2017-01-01 00:05:00|2017-01-01 00:01:00|-240 |-4.0 |
+-------------------+-------------------+--------+--------+
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(1,false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
only showing top 1 row
scala>
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53520559%2fhow-to-calculate-the-difference-in-time-between-2-row-in-spark-scala%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
0
You need to cast the column to timestamp and then do the diff calculation.
Check this out:
scala> val df = Seq(("1/01/2017 12:01:00 AM","1/1/2017 12:05:00 AM")).toDF("time1","time2")
df: org.apache.spark.sql.DataFrame = [time1: string, time2: string]
scala> val df2 = df.withColumn("time1",to_timestamp('time1,"d/MM/yyyy hh:mm:ss a")).withColumn("time2",to_timestamp('time2,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df2.printSchema
root
|-- time1: timestamp (nullable = true)
|-- time2: timestamp (nullable = true)
scala> df2.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
scala>
Update1:
scala> val df = Seq(("1/01/2017 12:01:00 AM"),("1/1/2017 12:05:00 AM")).toDF("timex")
df: org.apache.spark.sql.DataFrame = [timex: string]
scala> val df2 = df.withColumn("timex",to_timestamp('timex,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [timex: timestamp]
scala> df2.show
+-------------------+
| timex|
+-------------------+
|2017-01-01 00:01:00|
|2017-01-01 00:05:00|
+-------------------+
scala> val df3 = df2.alias("t1").join(df2.alias("t2"), $"t1.timex" =!= $"t2.timex", "leftOuter").toDF("time1","time2")
df3: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
|2017-01-01 00:05:00|2017-01-01 00:01:00|-240 |-4.0 |
+-------------------+-------------------+--------+--------+
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(1,false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
only showing top 1 row
scala>
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
add a comment |
0
You need to cast the column to timestamp and then do the diff calculation.
Check this out:
scala> val df = Seq(("1/01/2017 12:01:00 AM","1/1/2017 12:05:00 AM")).toDF("time1","time2")
df: org.apache.spark.sql.DataFrame = [time1: string, time2: string]
scala> val df2 = df.withColumn("time1",to_timestamp('time1,"d/MM/yyyy hh:mm:ss a")).withColumn("time2",to_timestamp('time2,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df2.printSchema
root
|-- time1: timestamp (nullable = true)
|-- time2: timestamp (nullable = true)
scala> df2.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
scala>
Update1:
scala> val df = Seq(("1/01/2017 12:01:00 AM"),("1/1/2017 12:05:00 AM")).toDF("timex")
df: org.apache.spark.sql.DataFrame = [timex: string]
scala> val df2 = df.withColumn("timex",to_timestamp('timex,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [timex: timestamp]
scala> df2.show
+-------------------+
| timex|
+-------------------+
|2017-01-01 00:01:00|
|2017-01-01 00:05:00|
+-------------------+
scala> val df3 = df2.alias("t1").join(df2.alias("t2"), $"t1.timex" =!= $"t2.timex", "leftOuter").toDF("time1","time2")
df3: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
|2017-01-01 00:05:00|2017-01-01 00:01:00|-240 |-4.0 |
+-------------------+-------------------+--------+--------+
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(1,false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
only showing top 1 row
scala>
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
add a comment |
0
0
0
You need to cast the column to timestamp and then do the diff calculation.
Check this out:
scala> val df = Seq(("1/01/2017 12:01:00 AM","1/1/2017 12:05:00 AM")).toDF("time1","time2")
df: org.apache.spark.sql.DataFrame = [time1: string, time2: string]
scala> val df2 = df.withColumn("time1",to_timestamp('time1,"d/MM/yyyy hh:mm:ss a")).withColumn("time2",to_timestamp('time2,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df2.printSchema
root
|-- time1: timestamp (nullable = true)
|-- time2: timestamp (nullable = true)
scala> df2.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
scala>
Update1:
scala> val df = Seq(("1/01/2017 12:01:00 AM"),("1/1/2017 12:05:00 AM")).toDF("timex")
df: org.apache.spark.sql.DataFrame = [timex: string]
scala> val df2 = df.withColumn("timex",to_timestamp('timex,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [timex: timestamp]
scala> df2.show
+-------------------+
| timex|
+-------------------+
|2017-01-01 00:01:00|
|2017-01-01 00:05:00|
+-------------------+
scala> val df3 = df2.alias("t1").join(df2.alias("t2"), $"t1.timex" =!= $"t2.timex", "leftOuter").toDF("time1","time2")
df3: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
|2017-01-01 00:05:00|2017-01-01 00:01:00|-240 |-4.0 |
+-------------------+-------------------+--------+--------+
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(1,false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
only showing top 1 row
scala>
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
You need to cast the column to timestamp and then do the diff calculation.
Check this out:
scala> val df = Seq(("1/01/2017 12:01:00 AM","1/1/2017 12:05:00 AM")).toDF("time1","time2")
df: org.apache.spark.sql.DataFrame = [time1: string, time2: string]
scala> val df2 = df.withColumn("time1",to_timestamp('time1,"d/MM/yyyy hh:mm:ss a")).withColumn("time2",to_timestamp('time2,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df2.printSchema
root
|-- time1: timestamp (nullable = true)
|-- time2: timestamp (nullable = true)
scala> df2.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
scala>
Update1:
scala> val df = Seq(("1/01/2017 12:01:00 AM"),("1/1/2017 12:05:00 AM")).toDF("timex")
df: org.apache.spark.sql.DataFrame = [timex: string]
scala> val df2 = df.withColumn("timex",to_timestamp('timex,"d/MM/yyyy hh:mm:ss a"))
df2: org.apache.spark.sql.DataFrame = [timex: timestamp]
scala> df2.show
+-------------------+
| timex|
+-------------------+
|2017-01-01 00:01:00|
|2017-01-01 00:05:00|
+-------------------+
scala> val df3 = df2.alias("t1").join(df2.alias("t2"), $"t1.timex" =!= $"t2.timex", "leftOuter").toDF("time1","time2")
df3: org.apache.spark.sql.DataFrame = [time1: timestamp, time2: timestamp]
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
|2017-01-01 00:05:00|2017-01-01 00:01:00|-240 |-4.0 |
+-------------------+-------------------+--------+--------+
scala> df3.withColumn("diff_sec",unix_timestamp('time2)-unix_timestamp('time1)).withColumn("diff_min",'diff_sec/60).show(1,false)
+-------------------+-------------------+--------+--------+
|time1 |time2 |diff_sec|diff_min|
+-------------------+-------------------+--------+--------+
|2017-01-01 00:01:00|2017-01-01 00:05:00|240 |4.0 |
+-------------------+-------------------+--------+--------+
only showing top 1 row
scala>
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
edited Nov 28 '18 at 15:04
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
answered Nov 28 '18 at 14:44
stack0114106stack0114106
4,8222423
4,8222423
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
add a comment |
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
the OP has changed the question..let me update the answer
– stack0114106
Nov 28 '18 at 15:02
add a comment |
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53520559%2fhow-to-calculate-the-difference-in-time-between-2-row-in-spark-scala%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Possible duplicate of Spark Scala: DateDiff of two columns by hour or minute
– vindev
Nov 28 '18 at 13:40
Can you add the datatype of column that shows time?
– Gofrette
Nov 28 '18 at 14:40