org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted Java Examples

The following examples show how to use org.apache.spark.streaming.scheduler.StreamingListenerBatchCompleted. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: ReceiverStreamListener.java From kafka-spark-consumer with Apache License 2.0

5 votes

@Override
public void onBatchCompleted(
        StreamingListenerBatchCompleted batchCompleted) {
    batchCompletedCount++;
    int queueSize = batchSubmittedCount - batchCompletedCount;
    boolean backPressureEnabled = (boolean) config._backpressureEnabled;
    if (backPressureEnabled) {
      long processingDelay =
              (Long) batchCompleted.batchInfo().processingDelay().get();
      long schedulingDelay =
              (Long) batchCompleted.batchInfo().schedulingDelay().get();
      //Skip first batch as it may pull very less records which can have wrong rate for next batch
      if(batchCompletedCount > 1) {
        // Get last batch fetch size
        int batchFetchSize = getFetchSize();
        LOG.info("Current Rate in ZooKeeper  : " + batchFetchSize);
        // Revise rate on last rate
        int newRate = controller.calculateRate(config, batchDuration,
                batchFetchSize, fillFreqMs, schedulingDelay, processingDelay);
        LOG.info("Modified Rate by Controller  : " + newRate);
        // Setting to Min Rate
        if (newRate < MIN_RATE) {
          newRate = MIN_RATE;
        }
        // Setting to Max Rate
        if (newRate > config._pollRecords) {
          newRate = config._pollRecords;
        }
        if (queueSize > THROTTLE_QUEUE) {
          LOG.warn("Controller rate not applied as waiting queue is greater than throttle queue");
        } else {
          Utils.setFetchRate(config, newRate);
        }
      }
    }
}

Example #2

Source File: SparkStreamingListener.java From cxf with Apache License 2.0

4 votes

@Override
public void onBatchCompleted(StreamingListenerBatchCompleted event) {
    System.out.println("Batch processing time in millisecs: " + (System.currentTimeMillis() - batchStartAt));

    streamOutput.setSparkBatchCompleted();
}

Example #3

Source File: AbstractJavaEsSparkStreamingTest.java From elasticsearch-hadoop with Apache License 2.0

4 votes

@Override
public void onBatchCompleted(StreamingListenerBatchCompleted batchCompleted) {
    // not implemented
}

Example #4

Source File: AbstractJavaEsSparkStreamingTest.java From elasticsearch-hadoop with Apache License 2.0

4 votes

@Override
public void onBatchCompleted(StreamingListenerBatchCompleted batchCompleted) {
    // not implemented
}