Java Code Examples for org.apache.samza.system.descriptors.InputDescriptor#getStreamId()

The following examples show how to use org.apache.samza.system.descriptors.InputDescriptor#getStreamId() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: TranslationContext.java    From beam with Apache License 2.0 6 votes vote down vote up 
public <OutT> void registerInputMessageStream(
    PValue pvalue,
    InputDescriptor<org.apache.samza.operators.KV<?, OpMessage<OutT>>, ?> inputDescriptor) {
  // we want to register it with the Samza graph only once per i/o stream
  final String streamId = inputDescriptor.getStreamId();
  if (registeredInputStreams.containsKey(streamId)) {
    MessageStream<OpMessage<OutT>> messageStream = registeredInputStreams.get(streamId);
    LOG.info(
        String.format(
            "Stream id %s has already been mapped to %s stream. Mapping %s to the same message stream.",
            streamId, messageStream, pvalue));
    registerMessageStream(pvalue, messageStream);

    return;
  }
  @SuppressWarnings("unchecked")
  final MessageStream<OpMessage<OutT>> typedStream =
      getValueStream(appDescriptor.getInputStream(inputDescriptor));

  registerMessageStream(pvalue, typedStream);
  registeredInputStreams.put(streamId, typedStream);
}
Example 2
Source File: StreamApplicationDescriptorImpl.java    From samza with Apache License 2.0 6 votes vote down vote up 
@Override
public <M> MessageStream<M> getInputStream(InputDescriptor<M, ?> inputDescriptor) {
  SystemDescriptor systemDescriptor = inputDescriptor.getSystemDescriptor();
  Optional<StreamExpander> expander = systemDescriptor.getExpander();
  if (expander.isPresent()) {
    return expander.get().apply(this, inputDescriptor);
  }

  // TODO: SAMZA-1841: need to add to the broadcast streams if inputDescriptor is for a broadcast stream
  addInputDescriptor(inputDescriptor);

  String streamId = inputDescriptor.getStreamId();
  Serde serde = inputDescriptor.getSerde();
  KV<Serde, Serde> kvSerdes = getOrCreateStreamSerdes(streamId, serde);
  boolean isKeyed = serde instanceof KVSerde;
  InputTransformer transformer = inputDescriptor.getTransformer().orElse(null);
  InputOperatorSpec inputOperatorSpec =
      OperatorSpecs.createInputOperatorSpec(streamId, kvSerdes.getKey(), kvSerdes.getValue(),
          transformer, isKeyed, this.getNextOpId(OpCode.INPUT, null));
  inputOperators.put(streamId, inputOperatorSpec);
  return new MessageStreamImpl(this, inputOperators.get(streamId));
}
Example 3
Source File: PortableTranslationContext.java    From beam with Apache License 2.0 5 votes vote down vote up 
/** Register an input stream with certain config id. */
public <T> void registerInputMessageStream(
    String id, InputDescriptor<KV<?, OpMessage<T>>, ?> inputDescriptor) {
  // we want to register it with the Samza graph only once per i/o stream
  final String streamId = inputDescriptor.getStreamId();
  if (registeredInputStreams.contains(streamId)) {
    return;
  }
  final MessageStream<OpMessage<T>> stream =
      appDescriptor.getInputStream(inputDescriptor).map(org.apache.samza.operators.KV::getValue);

  registerMessageStream(id, stream);
  registeredInputStreams.add(streamId);
}
Example 4
Source File: ApplicationDescriptorImpl.java    From samza with Apache License 2.0 5 votes vote down vote up 
final void addInputDescriptor(InputDescriptor inputDescriptor) {
  String streamId = inputDescriptor.getStreamId();
  Preconditions.checkState(!inputDescriptors.containsKey(streamId)
          || inputDescriptors.get(streamId) == inputDescriptor,
      String.format("Cannot add multiple input descriptors with the same streamId: %s", streamId));
  inputDescriptors.put(streamId, inputDescriptor);
  addSystemDescriptor(inputDescriptor.getSystemDescriptor());
}