Java Code Examples for org.apache.samza.system.IncomingMessageEnvelope#getSystemStreamPartition()
The following examples show how to use
org.apache.samza.system.IncomingMessageEnvelope#getSystemStreamPartition() .
You can vote up the ones you like or vote down the ones you don't like,
and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: TaskSideInputHandler.java From samza with Apache License 2.0 | 6 votes |
/** * Processes the incoming side input message envelope and updates the last processed offset for its SSP. * Synchronized inorder to be exclusive with flush(). * * @param envelope incoming envelope to be processed */ public synchronized void process(IncomingMessageEnvelope envelope) { SystemStreamPartition envelopeSSP = envelope.getSystemStreamPartition(); String envelopeOffset = envelope.getOffset(); for (String store: this.sspToStores.get(envelopeSSP)) { SideInputsProcessor storeProcessor = this.storeToProcessor.get(store); KeyValueStore keyValueStore = (KeyValueStore) this.taskSideInputStorageManager.getStore(store); Collection<Entry<?, ?>> entriesToBeWritten = storeProcessor.process(envelope, keyValueStore); // TODO: SAMZA-2255: optimize writes to side input stores for (Entry entry : entriesToBeWritten) { // If the key is null we ignore, if the value is null, we issue a delete, else we issue a put if (entry.getKey() != null) { if (entry.getValue() != null) { keyValueStore.put(entry.getKey(), entry.getValue()); } else { keyValueStore.delete(entry.getKey()); } } } } this.lastProcessedOffsets.put(envelopeSSP, envelopeOffset); }
Example 2
Source File: MultiFileHdfsReader.java From samza with Apache License 2.0 | 5 votes |
public IncomingMessageEnvelope readNext() { if (!hasNext()) { LOG.warn("Attempting to read more data when there aren't any. ssp=" + systemStreamPartition); return null; } // record the next offset before we read, so when the read fails and we reconnect, // we seek to the same offset that we try below curSingleFileOffset = curReader.nextOffset(); IncomingMessageEnvelope messageEnvelope = curReader.readNext(); // Copy everything except for the offset. Turn the single-file style offset into a multi-file one return new IncomingMessageEnvelope(messageEnvelope.getSystemStreamPartition(), getCurOffset(), messageEnvelope.getKey(), messageEnvelope.getMessage(), messageEnvelope.getSize(), messageEnvelope.getEventTime(), messageEnvelope.getArrivalTime()); }
Example 3
Source File: RunLoop.java From samza with Apache License 2.0 | 5 votes |
private boolean checkEndOfStream() { if (pendingEnvelopeQueue.size() == 1) { PendingEnvelope pendingEnvelope = pendingEnvelopeQueue.peek(); IncomingMessageEnvelope envelope = pendingEnvelope.envelope; if (envelope.isEndOfStream()) { SystemStreamPartition ssp = envelope.getSystemStreamPartition(); processingSspSet.remove(ssp); if (!hasIntermediateStreams) { pendingEnvelopeQueue.remove(); } } } return processingSspSet.isEmpty(); }
Example 4
Source File: FanOutTask.java From newsfeed with MIT License | 5 votes |
@Override @SuppressWarnings("unchecked") public void process(IncomingMessageEnvelope envelope, MessageCollector collector, TaskCoordinator coordinator) { String incomingStream = envelope.getSystemStreamPartition().getStream(); if (incomingStream.equals(NewsfeedConfig.FOLLOWS_STREAM.getStream())) { processFollowsEvent((Map<String, Object>) envelope.getMessage()); } else if (incomingStream.equals(NewsfeedConfig.MESSAGES_STREAM.getStream())) { processMessageEvent((Map<String, Object>) envelope.getMessage(), collector); } else { throw new IllegalStateException("Unexpected input stream: " + envelope.getSystemStreamPartition()); } }