org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker Java Examples

The following examples show how to use org.apache.beam.sdk.transforms.splittabledofn.OffsetRangeTracker. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: CSVContentProcessorDoFn.java    From dlp-dataflow-deidentification with Apache License 2.0 5 votes vote down vote up 
@ProcessElement
public void processElement(ProcessContext c, OffsetRangeTracker tracker) {
  for (long i = tracker.currentRestriction().getFrom(); tracker.tryClaim(i); ++i) {
    String fileName = c.element().getKey();
    String key = String.format("%s_%d", fileName, i);
    List<String> rows = c.element().getValue().stream().skip(1).collect(Collectors.toList());
    List<FieldId> headers =
        Arrays.stream(c.element().getValue().get(0).split(","))
            .map(header -> FieldId.newBuilder().setName(header).build())
            .collect(Collectors.toList());
    KV<Integer, Integer> lineRange = createStartEnd(rows.size(), i);
    int startOfLine = lineRange.getKey();
    int endOfLine = lineRange.getValue();

    List<String> lines = new ArrayList<>();

    for (int index = startOfLine - 1; index < endOfLine; index++) {

      lines.add(rows.get(index));
    }
    Table batchData = Util.createDLPTable(headers, lines);

    if (batchData.getRowsCount() > 0) {
      LOG.info(
          "Current Restriction From: {}, Current Restriction To: {}, StartofLine: {}, End Of Line {}, BatchData {}",
          tracker.currentRestriction().getFrom(),
          tracker.currentRestriction().getTo(),
          startOfLine,
          endOfLine,
          batchData.getRowsCount());
      c.output(KV.of(key, batchData));
      lines.clear();
    }
  }
}
Example #2
Source File: SplittableDoFnTest.java    From beam with Apache License 2.0 5 votes vote down vote up 
@NewTracker
public RestrictionTracker<OffsetRange, Long> newTracker(@Restriction OffsetRange restriction) {
  // Use a modified OffsetRangeTracker with only support for checkpointing.
  return new OffsetRangeTracker(restriction) {
    @Override
    public SplitResult<OffsetRange> trySplit(double fractionOfRemainder) {
      return super.trySplit(0);
    }
  };
}
Example #3
Source File: S3Import.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example #4
Source File: DLPTextToBigQueryStreaming.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example #5
Source File: CSVContentProcessorDoFn.java    From dlp-dataflow-deidentification with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example #6
Source File: DLPTextToBigQueryStreaming.java    From DataflowTemplates with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(@Restriction OffsetRange range) {
  return new OffsetRangeTracker(new OffsetRange(range.getFrom(), range.getTo()));
}
Example #7
Source File: FnApiDoFnRunnerTest.java    From beam with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public RestrictionTracker<OffsetRange, Long> newTracker(@Restriction OffsetRange restriction) {
  return new OffsetRangeTracker(restriction);
}
Example #8
Source File: Watch.java    From beam with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(@Restriction OffsetRange restriction) {
  return restriction.newTracker();
}
Example #9
Source File: SplittableParDoProcessFnTest.java    From beam with Apache License 2.0 4 votes vote down vote up 
@NewTracker
public OffsetRangeTracker newTracker(@Restriction OffsetRange range) {
  return new OffsetRangeTracker(range);
}