Java Code Examples for org.apache.beam.sdk.io.range.OffsetRange#getFrom()

The following examples show how to use org.apache.beam.sdk.io.range.OffsetRange#getFrom() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: PeriodicSequence.java    From beam with Apache License 2.0 5 votes vote down vote up 
@ProcessElement
public ProcessContinuation processElement(
    @Element SequenceDefinition srcElement,
    OutputReceiver<Instant> out,
    RestrictionTracker<OffsetRange, Long> restrictionTracker) {

  OffsetRange restriction = restrictionTracker.currentRestriction();
  Long interval = srcElement.durationMilliSec;
  Long nextOutput = restriction.getFrom() + interval;

  boolean claimSuccess = true;

  while (claimSuccess && Instant.ofEpochMilli(nextOutput).isBeforeNow()) {
    claimSuccess = restrictionTracker.tryClaim(nextOutput);
    if (claimSuccess) {
      Instant output = Instant.ofEpochMilli(nextOutput);
      out.outputWithTimestamp(output, output);
      nextOutput = nextOutput + interval;
    }
  }

  ProcessContinuation continuation = ProcessContinuation.stop();
  if (claimSuccess) {
    Duration offset = new Duration(Instant.now(), Instant.ofEpochMilli(nextOutput));
    continuation = ProcessContinuation.resume().withResumeDelay(offset);
  }
  return continuation;
}
Example 2
Source File: S3Import.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example 3
Source File: DLPTextToBigQueryStreaming.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example 4
Source File: CSVContentProcessorDoFn.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example 5
Source File: DLPTextToBigQueryStreaming.java    From DataflowTemplates with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(@Restriction OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example 6
Source File: PCollectionViewsTest.java    From beam with Apache License 2.0 4 votes vote down vote up 
private static void assertNonEmptyRangesAndPositions(
    Iterable<OffsetRange> ranges,
    Map<OffsetRange, Integer> nonOverlappingRangesToNumElementsPerPosition) {
  for (Map.Entry<OffsetRange, Integer> entry :
      nonOverlappingRangesToNumElementsPerPosition.entrySet()) {
    assertNotEquals(0, (int) entry.getValue());
  }

  ListMultimap<Long, Integer> positions = ArrayListMultimap.create();
  for (OffsetRange range : ranges) {
    for (long i = range.getFrom(); i < range.getTo(); ++i) {
      positions.put(i, 0);
    }
  }

  int position = 0;
  for (Long key : new TreeSet<>(positions.keySet())) {
    int size = positions.get(key).size();
    positions.replaceValues(
        key, Lists.newArrayList(IntStream.range(position, position + size).iterator()));
    position += size;
  }

  for (int i = 0; i < position; ++i) {
    KV<Long, Integer> computedPosition =
        computePositionForIndex(nonOverlappingRangesToNumElementsPerPosition, i);
    assertEquals(
        i, (int) positions.get(computedPosition.getKey()).get(computedPosition.getValue()));
  }

  assertThrows(
      IndexOutOfBoundsException.class,
      () -> computePositionForIndex(nonOverlappingRangesToNumElementsPerPosition, -1));

  int totalNumberOfElements =
      computeTotalNumElements(nonOverlappingRangesToNumElementsPerPosition);
  assertEquals(position, totalNumberOfElements);
  assertThrows(
      IndexOutOfBoundsException.class,
      () ->
          computePositionForIndex(
              nonOverlappingRangesToNumElementsPerPosition, totalNumberOfElements));
}