org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted Java Examples

The following examples show how to use org.apache.spark.streaming.scheduler.StreamingListenerBatchSubmitted. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: ReceiverStreamListener.java From kafka-spark-consumer with Apache License 2.0

5 votes

@Override
public void onBatchSubmitted(StreamingListenerBatchSubmitted arg0) {
  batchSubmittedCount++;
  int queueSize = batchSubmittedCount - batchCompletedCount;
  if (queueSize > THROTTLE_QUEUE) {
    LOG.warn("stop consumer as pending queue {} greater than configured limit {}",queueSize, THROTTLE_QUEUE);
    //Set fetch size to 10 messages to throttle the consumer
    Utils.setFetchRate(config,MIN_RATE);
  } else if(!config._backpressureEnabled) {
    Utils.setFetchRate(config, config._pollRecords);
  }
}

Example #2

Source File: SparkStreamingListener.java From cxf with Apache License 2.0

4 votes

@Override
public void onBatchSubmitted(StreamingListenerBatchSubmitted event) {
}

Example #3

Source File: AbstractJavaEsSparkStreamingTest.java From elasticsearch-hadoop with Apache License 2.0

4 votes

@Override
public void onBatchSubmitted(StreamingListenerBatchSubmitted batchSubmitted) {
    // not implemented
}

Example #4

Source File: AbstractJavaEsSparkStreamingTest.java From elasticsearch-hadoop with Apache License 2.0

4 votes

@Override
public void onBatchSubmitted(StreamingListenerBatchSubmitted batchSubmitted) {
    // not implemented
}