Java Code Examples for org.apache.samza.system.SystemStream#getStream()

The following examples show how to use org.apache.samza.system.SystemStream#getStream() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: TransactionalStateTaskRestoreManager.java    From samza with Apache License 2.0 6 votes vote down vote up 
/**
 * Get offset metadata for each changelog SSP for this task. A task may have multiple changelog streams
 * (e.g., for different stores), but will have the same partition for all of them.
 */
@VisibleForTesting
static Map<SystemStreamPartition, SystemStreamPartitionMetadata> getCurrentChangelogOffsets(
    TaskModel taskModel, Map<String, SystemStream> storeChangelogs, SSPMetadataCache sspMetadataCache) {
  Map<SystemStreamPartition, SystemStreamPartitionMetadata> changelogOffsets = new HashMap<>();

  Partition changelogPartition = taskModel.getChangelogPartition();
  for (Map.Entry<String, SystemStream> storeChangelog : storeChangelogs.entrySet()) {
    SystemStream changelog = storeChangelog.getValue();
    SystemStreamPartition changelogSSP = new SystemStreamPartition(
        changelog.getSystem(), changelog.getStream(), changelogPartition);
    SystemStreamPartitionMetadata metadata = sspMetadataCache.getMetadata(changelogSSP);
    changelogOffsets.put(changelogSSP, metadata);
  }

  LOG.info("Got current changelog offsets for taskName: {} as: {}", taskModel.getTaskName(), changelogOffsets);
  return changelogOffsets;
}
Example 2
Source File: ConfigBasedSspGrouperFactory.java    From samza with Apache License 2.0 5 votes vote down vote up 
private Set<Integer> getPartitions(SystemStream systemStream) {
  String streamName = systemStream.getStream();

  if (!streamPartitionsMap.containsKey(streamName)) {
    String partitions = config.get(String.format(CONFIG_STREAM_PARTITIONS, streamName));
    streamPartitionsMap.put(streamName, Arrays.stream(partitions.split(CFG_PARTITIONS_DELIMITER))
        .map(Integer::parseInt)
        .collect(Collectors.toSet()));
  }
  return streamPartitionsMap.get(streamName);
}
Example 3
Source File: DiagnosticsUtil.java    From samza with Apache License 2.0 5 votes vote down vote up 
public static void createDiagnosticsStream(Config config) {
  if (!new JobConfig(config).getDiagnosticsEnabled()) {
    return;
  }
  // if diagnostics is enabled, create diagnostics stream if it doesnt exist

  SystemAdmins systemAdmins = new SystemAdmins(config);
  String diagnosticsSystemStreamName = new MetricsConfig(config)
      .getMetricsSnapshotReporterStream(MetricsConfig.METRICS_SNAPSHOT_REPORTER_NAME_FOR_DIAGNOSTICS)
      .orElseThrow(() -> new ConfigException("Missing required config: " +
          String.format(MetricsConfig.METRICS_SNAPSHOT_REPORTER_STREAM,
              MetricsConfig.METRICS_SNAPSHOT_REPORTER_NAME_FOR_DIAGNOSTICS)));

  SystemStream diagnosticsSystemStream = StreamUtil.getSystemStreamFromNames(diagnosticsSystemStreamName);
  SystemAdmin diagnosticsSysAdmin = systemAdmins.getSystemAdmin(diagnosticsSystemStream.getSystem());
  StreamSpec diagnosticsStreamSpec = new StreamSpec(DIAGNOSTICS_STREAM_ID, diagnosticsSystemStream.getStream(),
      diagnosticsSystemStream.getSystem(), new StreamConfig(config).getStreamProperties(DIAGNOSTICS_STREAM_ID));

  log.info("Creating diagnostics stream {}", diagnosticsSystemStream.getStream());
  diagnosticsSysAdmin.start();

  if (diagnosticsSysAdmin.createStream(diagnosticsStreamSpec)) {
    log.info("Created diagnostics stream {}", diagnosticsSystemStream.getStream());
  } else {
    log.info("Diagnostics stream {} already exists", diagnosticsSystemStream.getStream());
  }

  diagnosticsSysAdmin.stop();
}
Example 4
Source File: AvroSchemaGenRelConverter.java    From samza with Apache License 2.0 4 votes vote down vote up 
public AvroSchemaGenRelConverter(SystemStream systemStream, AvroRelSchemaProvider schemaProvider, Config config) {
  super(systemStream, schemaProvider, config);
  streamName = systemStream.getStream();
}
Example 5
Source File: StreamUtil.java    From samza with Apache License 2.0 2 votes vote down vote up 
/**
 * Returns the period separated system stream name for the provided {@code systemStream}. For
 * example, SystemStream("kafka", "topic") would return "kafka.topic".
 *
 * @param systemStream the {@link SystemStream} to get the name for
 * @return the system stream name
 */
public static String getNameFromSystemStream(SystemStream systemStream) {
  return systemStream.getSystem() + "." + systemStream.getStream();
}