Java Code Examples for org.apache.spark.rdd.RDD#toJavaRDD()

The following examples show how to use org.apache.spark.rdd.RDD#toJavaRDD() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: GraphXGraphGenerator.java    From rya with Apache License 2.0 8 votes vote down vote up 
public Graph<RyaTypeWritable, RyaTypeWritable> createGraph(SparkContext sc, Configuration conf) throws IOException, AccumuloSecurityException{
    StorageLevel storageLvl1 = StorageLevel.MEMORY_ONLY();
    StorageLevel storageLvl2 = StorageLevel.MEMORY_ONLY();
    ClassTag<RyaTypeWritable> RTWTag = ClassTag$.MODULE$.apply(RyaTypeWritable.class);
    RyaTypeWritable rtw = null;
    RDD<Tuple2<Object, RyaTypeWritable>> vertexRDD = getVertexRDD(sc, conf);

    RDD<Tuple2<Object, Edge>> edgeRDD = getEdgeRDD(sc, conf);
    JavaRDD<Tuple2<Object, Edge>> jrddTuple = edgeRDD.toJavaRDD();
    JavaRDD<Edge<RyaTypeWritable>> jrdd = jrddTuple.map(tuple -> tuple._2);

    RDD<Edge<RyaTypeWritable>> goodERDD = JavaRDD.toRDD(jrdd);

    return Graph.apply(vertexRDD, goodERDD, rtw, storageLvl1, storageLvl2, RTWTag, RTWTag);
}
Example 2
Source File: BasicStatisticsComputer.java    From DDF with Apache License 2.0 5 votes vote down vote up 
@SuppressWarnings("unchecked")
@Override
public Summary[] getSummaryImpl() throws DDFException {
  RDD<Object[]> rdd = (RDD<Object[]>) this.getDDF().getRepresentationHandler().get(RDD.class, Object[].class);

  JavaRDD<Object[]> data = rdd.toJavaRDD();
  Summary[] stats = data.map(new GetSummaryMapper()).reduce(new GetSummaryReducer());
  return stats;
}
Example 3
Source File: SparkDDF.java    From DDF with Apache License 2.0 5 votes vote down vote up 
@SuppressWarnings({ "rawtypes", "unchecked" })
public IGetResult getJavaRDD(Class<?>... acceptableUnitTypes) throws DDFException {
  IGetResult result = this.getRDD(acceptableUnitTypes);
  RDD<?> rdd = (RDD<?>) result.getObject();
  Class<?> unitType = result.getTypeSpecs()[1];

  return new GetResult(rdd.toJavaRDD(), unitType);
}
Example 4
Source File: SparkDDF.java    From DDF with Apache License 2.0 4 votes vote down vote up 
public <T> JavaRDD<T> getJavaRDD(Class<T> unitType) throws DDFException {
  RDD<T> rdd = this.getRDD(unitType);
  return rdd.toJavaRDD();
}