Java Code Examples for org.apache.beam.sdk.io.range.OffsetRange#split()

The following examples show how to use org.apache.beam.sdk.io.range.OffsetRange#split() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: HL7v2IO.java    From beam with Apache License 2.0 6 votes vote down vote up 
@SplitRestriction
public void split(@Restriction OffsetRange timeRange, OutputReceiver<OffsetRange> out) {
  List<OffsetRange> splits =
      timeRange.split(initialSplitDuration.getMillis(), DEFAULT_MIN_SPLIT_DURATION.getMillis());
  Instant from = Instant.ofEpochMilli(timeRange.getFrom());
  Instant to = Instant.ofEpochMilli(timeRange.getTo());
  Duration totalDuration = new Duration(from, to);
  LOG.info(
      String.format(
          "splitting initial sendTime restriction of [minSendTime, now): [%s,%s), "
              + "or [%s, %s). \n"
              + "total days: %s \n"
              + "into %s splits. \n"
              + "Last split: %s",
          from,
          to,
          timeRange.getFrom(),
          timeRange.getTo(),
          totalDuration.getStandardDays(),
          splits.size(),
          splits.get(splits.size() - 1).toString()));

  for (OffsetRange s : splits) {
    out.output(s);
  }
}
Example 2
Source File: S3Import.java    From dlp-dataflow-deidentification with Apache License 2.0 5 votes vote down vote up 
@SplitRestriction
public void splitRestriction(
    KV<String, ReadableFile> file, OffsetRange range, OutputReceiver<OffsetRange> out) {

  for (final OffsetRange p : range.split(1, 1)) {
    out.output(p);
  }
}
Example 3
Source File: DLPTextToBigQueryStreaming.java    From dlp-dataflow-deidentification with Apache License 2.0 5 votes vote down vote up 
/**
 * SDF needs to define a @SplitRestriction method that can split the intital restricton to a
 * number of smaller restrictions. For example: a intital rewstriction of (x, N) as input and
 * produces pairs (x, 0), (x, 1), …, (x, N-1) as output.
 */
@SplitRestriction
public void splitRestriction(
    KV<String, ReadableFile> csvFile, OffsetRange range, OutputReceiver<OffsetRange> out) {
  /** split the initial restriction by 1 */
  for (final OffsetRange p : range.split(1, 1)) {
    out.output(p);
  }
}
Example 4
Source File: CSVContentProcessorDoFn.java    From dlp-dataflow-deidentification with Apache License 2.0 5 votes vote down vote up 
@SplitRestriction
public void splitRestriction(
    KV<String, List<String>> contents, OffsetRange range, OutputReceiver<OffsetRange> out) {
  for (final OffsetRange p : range.split(1, 1)) {
    out.output(p);
  }
}
Example 5
Source File: DLPTextToBigQueryStreaming.java    From DataflowTemplates with Apache License 2.0 5 votes vote down vote up 
/**
 * SDF needs to define a @SplitRestriction method that can split the intital restricton to a
 * number of smaller restrictions. For example: a intital rewstriction of (x, N) as input and
 * produces pairs (x, 0), (x, 1), …, (x, N-1) as output.
 */
@SplitRestriction
public void splitRestriction(
    @Element KV<String, ReadableFile> csvFile,@Restriction OffsetRange range, OutputReceiver<OffsetRange> out) {
  /** split the initial restriction by 1 */
  for (final OffsetRange p : range.split(1, 1)) {
    out.output(p);
  }
}