Btukfyl

How does one create an RDD filled with values from an array say (0,1) - filling random 1000 values as 1 and remaining 0.

I know I can filter and do this but it won't be random. I want it to be as random as possible

var populationMatrix = new IndexedRowMatrix(RandomRDDs.uniformVectorRDD(sc, populationSize, chromosomeLength)

I was exploring random RDDs in spark but could find something that meets my needs .

Not really sure if this is what you are looking for, but with this code you are able to create an RDD array with random numbers between 0 and 1s:

import scala.util.Random



val arraySize = 15 // Total number of elements that you want

val numberOfOnes = 10 // From that total, how many do you want to be ones

val listOfOnes = List.fill(numberOfOnes)(1) // List of 1s

val listOfZeros = List.fill(arraySize - numberOfOnes)(0) // Rest list of 0s

val listOfOnesAndZeros = listOfOnes ::: listOfZeros // Merge lists

val randomList = Random.shuffle(listOfOnesAndZeros) // Random shuffle

val randomRDD = sc.parallelize(randomList) // RDD creation

randomRDD.collect() // Array[Int] = Array(1, 0, 0, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 1, 1)

Or, if you want to use only RDDs:

val arraySize = 15

val numberOfOnes = 10



val rddOfOnes = spark.range(numberOfOnes).map(_ => 1).rdd

val rddOfZeros = spark.range(arraySize - numberOfOnes).map(_ => 0).rdd

val rddOfOnesAndZeros = rddOfOnes ++ rddOfZeros

val shuffleResult = rddOfOnesAndZeros.mapPartitions(iter => {

  val rng = new scala.util.Random()

  iter.map((rng.nextInt, _))

}).partitionBy(new org.apache.spark.HashPartitioner(rddOfOnesAndZeros.partitions.size)).values



shuffleResult.collect() // Array[Int] = Array(0, 1, 1, 0, 1, 1, 1, 1, 1, 0, 0, 0, 1, 1, 1)

Let me know if it was what you need it.