Java Code Examples for org.apache.kafka.clients.consumer.ConsumerRecord#serializedKeySize()

The following examples show how to use org.apache.kafka.clients.consumer.ConsumerRecord#serializedKeySize() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: AbstractKafkaBasedConnectorTask.java    From brooklin with BSD 2-Clause "Simplified" License 5 votes vote down vote up 
/**
 * Translate the Kafka consumer records if necessary and send the batch of records to destination.
 * @param records the Kafka consumer records
 * @param readTime the instant the records were successfully polled from the Kafka source
 */
protected void translateAndSendBatch(ConsumerRecords<?, ?> records, Instant readTime) {
  // iterate through each topic partition one at a time, for better isolation
  for (TopicPartition topicPartition : records.partitions()) {
    for (ConsumerRecord<?, ?> record : records.records(topicPartition)) {
      try {
        boolean partitionPaused;
        boolean sendFailure;
        synchronized (_sendFailureTopicPartitionExceptionMap) {
          partitionPaused = _autoPausedSourcePartitions.containsKey(topicPartition);
          sendFailure = _sendFailureTopicPartitionExceptionMap.containsKey(topicPartition);
        }
        if (partitionPaused || sendFailure) {
          _logger.warn("Abort sending for {}, auto-paused: {}, send failure: {}, rewind offset", topicPartition,
              partitionPaused, sendFailure);
          seekToLastCheckpoint(Collections.singleton(topicPartition));
          break;
        } else {
          DatastreamProducerRecord datastreamProducerRecord = translate(record, readTime);
          int numBytes = record.serializedKeySize() + record.serializedValueSize();
          sendDatastreamProducerRecord(datastreamProducerRecord, topicPartition, numBytes, null);
        }
      } catch (Exception e) {
        _logger.warn(String.format("Got exception while sending record %s, exception: ", record), e);
        rewindAndPausePartitionOnException(topicPartition, e);
        // skip other messages for this partition, but can continue processing other partitions
        break;
      }
    }
  }
}
Example 2
Source File: DBusConsumerRecord.java    From DBus with Apache License 2.0 4 votes vote down vote up 
public DBusConsumerRecord(ConsumerRecord<K, V> record) {
    this(record.topic(), record.partition(), record.offset(), record.timestamp(), record.timestampType(),
            record.checksum(), record.serializedKeySize(), record.serializedValueSize(), record.key(), record.value());
}
Example 3
Source File: KafkaConsumerProxy.java    From samza with Apache License 2.0 2 votes vote down vote up 
/**
 * Protected to help extensions of this class build {@link IncomingMessageEnvelope}s.
 * @param r consumer record to size
 * @return the size of the serialized record
 */
protected int getRecordSize(ConsumerRecord<K, V> r) {
  int keySize = (r.key() == null) ? 0 : r.serializedKeySize();
  return keySize + r.serializedValueSize();
}