Java Code Examples for org.apache.spark.serializer.SerializerInstance#deserialize()

The following examples show how to use org.apache.spark.serializer.SerializerInstance#deserialize() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: SparkFrontendUtils.java    From incubator-nemo with Apache License 2.0 6 votes vote down vote up 
/**
 * Converts a {@link Function1} to a corresponding {@link Function}.
 * <p>
 * Here, we use the Spark 'JavaSerializer' to facilitate debugging in the future.
 * TODO #205: RDD Closure with Broadcast Variables Serialization Bug
 *
 * @param scalaFunction the scala function to convert.
 * @param <I>           the type of input.
 * @param <O>           the type of output.
 * @return the converted Java function.
 */
public static <I, O> Function<I, O> toJavaFunction(final Function1<I, O> scalaFunction) {
  // This 'JavaSerializer' from Spark provides a human-readable NotSerializableException stack traces,
  // which can be useful when addressing this problem.
  // Other toJavaFunction can also use this serializer when debugging.
  final ClassTag<Function1<I, O>> classTag = ClassTag$.MODULE$.apply(scalaFunction.getClass());
  final byte[] serializedFunction = new JavaSerializer().newInstance().serialize(scalaFunction, classTag).array();

  return new Function<I, O>() {
    private Function1<I, O> deserializedFunction;

    @Override
    public O call(final I v1) throws Exception {
      if (deserializedFunction == null) {
        // TODO #205: RDD Closure with Broadcast Variables Serialization Bug
        final SerializerInstance js = new JavaSerializer().newInstance();
        deserializedFunction = js.deserialize(ByteBuffer.wrap(serializedFunction), classTag);
      }
      return deserializedFunction.apply(v1);
    }
  };
}
Example 2
Source File: TestNd4jKryoSerialization.java    From nd4j with Apache License 2.0 5 votes vote down vote up 
private <T> void testSerialization(T in, SerializerInstance si) {
        ByteBuffer bb = si.serialize(in, null);
        T deserialized = (T)si.deserialize(bb, null);

//        assertEquals(in, deserialized);
        boolean equals = in.equals(deserialized);
        assertTrue(in.getClass() + "\t" + in.toString(), equals);
    }
Example 3
Source File: TestNd4jKryoSerialization.java    From deeplearning4j with Apache License 2.0 5 votes vote down vote up 
private <T> void testSerialization(T in, SerializerInstance si) {
        ByteBuffer bb = si.serialize(in, null);
        T deserialized = (T)si.deserialize(bb, null);

//        assertEquals(in, deserialized);
        boolean equals = in.equals(deserialized);
        assertTrue(in.getClass() + "\t" + in.toString(), equals);
    }
Example 4
Source File: TestKryo.java    From deeplearning4j with Apache License 2.0 5 votes vote down vote up 
private <T> void testSerialization(T in, SerializerInstance si) {
    ByteBuffer bb = si.serialize(in, null);
    T deserialized = (T)si.deserialize(bb, null);

    boolean equals = in.equals(deserialized);
    assertTrue(in.getClass() + "\t" + in.toString(), equals);
}
Example 5
Source File: TestKryoSerialization.java    From DataVec with Apache License 2.0 4 votes vote down vote up 
private <T> T serDe(T in, SerializerInstance si){
    ByteBuffer bb = si.serialize(in, null);
    return (T)si.deserialize(bb, null);
}
Example 6
Source File: TestKryoSerialization.java    From deeplearning4j with Apache License 2.0 4 votes vote down vote up 
private <T> T serDe(T in, SerializerInstance si){
    ByteBuffer bb = si.serialize(in, null);
    return (T)si.deserialize(bb, null);
}
Example 7
Source File: SparkTestUtils.java    From gatk with BSD 3-Clause "New" or "Revised" License 3 votes vote down vote up 
/**
 * Takes an input object and returns the value of the object after it has been serialized and then deserialized in Kryo.
 * Requires the class of the input object as a parameter because it's not generally possible to get the class of a
 * generified method parameter with reflection.
 *
 * @param input instance of inputClazz.  Never {@code null}
 * @param inputClazz class to cast input
 * @param conf Spark configuration to test
 * @param <T> class to attempt.  Same or subclass of inputClazz
 * @return serialized and deserialized instance of input.  Throws exception if serialization round trip fails.
 */
public static <T> T roundTripInKryo(final T input, final Class<?> inputClazz, final SparkConf conf) {
    Utils.nonNull(input);
    final KryoSerializer kryoSerializer = new KryoSerializer(conf);
    final SerializerInstance sparkSerializer = kryoSerializer.newInstance();
    final ClassTag<T> tag = ClassTag$.MODULE$.apply(inputClazz);
    return sparkSerializer.deserialize(sparkSerializer.serialize(input, tag), tag);
}