Btukfyl

I'm running spark-shell to compare 2 csv files. Each file has the same number of columns and all have 600,000 rows. I'm expecting the 2 files have all the same rows. Here is my script.

val a =

  spark

    .read

    .option("header", "true")

    .option("delimiter", "|")

    .csv("/tmp/1.csv")

    .drop("unwanted_column").

    .cache()



val b = 

   spark

     .read

     .option("header", "true")

     .option("delimiter", "|")

     .csv("/tmp/2.csv")

     .drop("unwanted_column")

     .cache()



val c = a.join(b, Seq("id", "year"), "left_outer").cache()



c.count() // this is returning 600,000

Now I'm trying to find out the difference by randomly picking a line with the same id and year in 2 datasets a and b.

val a1 = a.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()



val b1 = b.filter(i => i.get(0).equals("1") && i.get(1).equals("2016")).first()

Then I try to compare each column in a1 and b1.

(0 to (a1.length -1)).foreach { i =>

  if (a1.getString(i) != null && !a1.getString(i).equals(b1.getString(i))) {

    System.out.println(i + " = " + a1.getString(i) + " = " + b1.getString(i))

  }

}

It didn't print anything. In other words, there is no difference.

I can't tell why c.count() is returning 600,000 like that.

Sorry guys, I guess it was my fault. Actually I was after a.subtract(b). My purpose is to find out the difference between a and b. I was confused about left_outer join.