org.apache.beam.sdk.transforms.ProcessFunction Java Examples

The following examples show how to use org.apache.beam.sdk.transforms.ProcessFunction. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: TypeDescriptors.java    From beam with Apache License 2.0 5 votes vote down vote up 
/**
 * Returns a type descriptor for the input of the given {@link ProcessFunction}, subject to Java
 * type erasure: may contain unresolved type variables if the type was erased.
 */
public static <InputT, OutputT> TypeDescriptor<InputT> inputOf(
    ProcessFunction<InputT, OutputT> fn) {
  return extractFromTypeParameters(
      fn,
      ProcessFunction.class,
      new TypeVariableExtractor<ProcessFunction<InputT, OutputT>, InputT>() {});
}
Example #2
Source File: TypeDescriptors.java    From beam with Apache License 2.0 5 votes vote down vote up 
/**
 * Returns a type descriptor for the output of the given {@link ProcessFunction}, subject to Java
 * type erasure: may contain unresolved type variables if the type was erased.
 */
public static <InputT, OutputT> TypeDescriptor<OutputT> outputOf(
    ProcessFunction<InputT, OutputT> fn) {
  return extractFromTypeParameters(
      fn,
      ProcessFunction.class,
      new TypeVariableExtractor<ProcessFunction<InputT, OutputT>, OutputT>() {});
}
Example #3
Source File: TypeDescriptors.java    From beam with Apache License 2.0 4 votes vote down vote up 
/** Binary compatibility adapter for {@link #inputOf(ProcessFunction)}. */
public static <InputT, OutputT> TypeDescriptor<InputT> inputOf(
    SerializableFunction<InputT, OutputT> fn) {
  return inputOf((ProcessFunction<InputT, OutputT>) fn);
}
Example #4
Source File: TypeDescriptors.java    From beam with Apache License 2.0 4 votes vote down vote up 
/** Binary compatibility adapter for {@link #outputOf(ProcessFunction)}. */
public static <InputT, OutputT> TypeDescriptor<OutputT> outputOf(
    SerializableFunction<InputT, OutputT> fn) {
  return outputOf((ProcessFunction<InputT, OutputT>) fn);
}
Example #5
Source File: Transforms.java    From nomulus with Apache License 2.0 4 votes vote down vote up 
private BackupFileReader(ProcessFunction<ReadableFile, Iterator<VersionedEntity>> reader) {
  this.reader = reader;
}
Example #6
Source File: ParseJsons.java    From beam with Apache License 2.0 3 votes vote down vote up 
/**
 * Returns a new {@link ParseJsonsWithFailures} transform that catches exceptions raised while
 * parsing elements, passing the raised exception instance and the input element being processed
 * through the given {@code exceptionHandler} and emitting the result to a failure collection.
 * It is supposed to be used along with {@link ParseJsons#exceptionsInto(TypeDescriptor)} and
 * get lambda function as exception handler.
 *
 * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
 * WithFailures.Result}.
 *
 * <p>Example usage:
 *
 * <pre>{@code
 * WithFailures.Result<PCollection<MyPojo>, KV<String, Map<String, String>>> result =
 *     json.apply(
 *         ParseJsons.of(MyPojo.class)
 *             .exceptionsInto(
 *                 TypeDescriptors.kvs(TypeDescriptors.strings(), TypeDescriptors.strings()))
 *             .exceptionsVia(
 *                 f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
 *
 * PCollection<MyPojo> output = result.output(); // valid POJOs
 * PCollection<KV<String, Map<String, String>>> failures = result.failures();
 * }</pre>
 */
public ParseJsonsWithFailures<FailureT> exceptionsVia(
    ProcessFunction<WithFailures.ExceptionElement<String>, FailureT> exceptionHandler) {
  return new ParseJsonsWithFailures<>(
      new InferableFunction<WithFailures.ExceptionElement<String>, FailureT>(
          exceptionHandler) {},
      failureType);
}
Example #7
Source File: AsJsons.java    From beam with Apache License 2.0 3 votes vote down vote up 
/**
 * Returns a new {@link AsJsonsWithFailures} transform that catches exceptions raised while
 * writing JSON elements, passing the raised exception instance and the input element being
 * processed through the given {@code exceptionHandler} and emitting the result to a failure
 * collection. It is supposed to be used along with {@link
 * AsJsons#exceptionsInto(TypeDescriptor)} and get lambda function as exception handler.
 *
 * <p>See {@link WithFailures} documentation for usage patterns of the returned {@link
 * WithFailures.Result}.
 *
 * <p>Example usage:
 *
 * <pre>{@code
 * WithFailures.Result<PCollection<String>, KV<MyPojo, Map<String, String>>> result =
 *     pojos.apply(
 *         AsJsons.of(MyPojo.class)
 *             .exceptionsInto(
 *                 TypeDescriptors.kvs(
 *                     TypeDescriptor.of(MyPojo.class), TypeDescriptors.strings()))
 *             .exceptionsVia(
 *                 f -> KV.of(f.element(), f.exception().getClass().getCanonicalName())));
 *
 * PCollection<String> output = result.output(); // valid json elements
 * PCollection<KV<MyPojo, Map<String, String>>> failures = result.failures();
 * }</pre>
 */
public AsJsonsWithFailures<FailureT> exceptionsVia(
    ProcessFunction<WithFailures.ExceptionElement<InputT>, FailureT> exceptionHandler) {
  return new AsJsonsWithFailures<>(
      new InferableFunction<WithFailures.ExceptionElement<InputT>, FailureT>(
          exceptionHandler) {},
      failureType);
}