Java Code Examples for org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage#getAttribute()

The following examples show how to use org.apache.beam.sdk.io.gcp.pubsub.PubsubMessage#getAttribute() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example 1

Source File: RepublishPerDocType.java From gcp-ingestion with Mozilla Public License 2.0

5 votes

@Override
public int partitionFor(PubsubMessage message, int numPartitions) {
  message = PubsubConstraints.ensureNonNull(message);
  String docType = message.getAttribute("document_type");
  String namespace = message.getAttribute("document_namespace");
  for (int i = 0; i < destinations.size(); i++) {
    if (destinations.get(i).matches(namespace, docType)) {
      return i;
    }
  }
  // The last partition catches docTypes that aren't configured to have a destination;
  // these will be ignored.
  return numPartitions - 1;
}

Example 2

Source File: RepublishPerNamespace.java From gcp-ingestion with Mozilla Public License 2.0

5 votes

@Override
public int partitionFor(PubsubMessage message, int numPartitions) {
  message = PubsubConstraints.ensureNonNull(message);
  String namespace = message.getAttribute("document_namespace");
  for (int i = 0; i < destinations.size(); i++) {
    if (destinations.get(i).matches(namespace)) {
      return i;
    }
  }
  // The last partition catches docTypes that aren't configured to have a destination;
  // these will be ignored.
  return numPartitions - 1;
}

Example 3

Source File: RepublishPerChannel.java From gcp-ingestion with Mozilla Public License 2.0

5 votes

@Override
public int partitionFor(PubsubMessage message, int numPartitions) {
  message = PubsubConstraints.ensureNonNull(message);
  String channel = message.getAttribute("normalized_channel");
  for (int i = 0; i < destinations.size(); i++) {
    if (destinations.get(i).matches(channel)) {
      return i;
    }
  }
  // The last partition catches docTypes that aren't configured to have a destination;
  // these will be ignored.
  return numPartitions - 1;
}

Example 4

Source File: PubsubToPubsub.java From DataflowTemplates with Apache License 2.0

5 votes

@ProcessElement
public void processElement(ProcessContext context) {

  INPUT_COUNTER.inc();
  if (!this.doFilter) {

    // Filter is not enabled
    writeOutput(context, context.element());
  } else {

    PubsubMessage message = context.element();
    String extractedValue = message.getAttribute(this.inputFilterKey);

    if (this.isNullFilterValue) {

      if (extractedValue == null) {
        // If we are filtering for null and the extracted value is null, we forward
        // the message.
        writeOutput(context, message);
      }

    } else {

      if (extractedValue != null
          && this.inputFilterValueRegex.matcher(extractedValue).matches()) {
        // If the extracted value is not null and it matches the filter,
        // we forward the message.
        writeOutput(context, message);
      }
    }
  }
}