org.apache.kafka.connect.runtime.Herder Java Examples

The following examples show how to use org.apache.kafka.connect.runtime.Herder. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: ConnectStandalone.java    From mongo-kafka with Apache License 2.0 6 votes vote down vote up 
@SuppressWarnings({"unchecked", "rawtypes"})
void addConnector(final String name, final Properties properties) {
  FutureCallback<Herder.Created<ConnectorInfo>> cb =
      new FutureCallback<>(
          (error, info) -> {
            if (error != null) {
              LOGGER.error("Failed to create job for {}", properties);
            } else {
              LOGGER.info("Created connector {}", info.result().name());
            }
          });
  try {
    herder.putConnectorConfig(name, (Map) properties, true, cb);
    cb.get();
    sleep(1000);
  } catch (Exception e) {
    LOGGER.error("Failed to add connector for {}", properties);
    throw new ConnectorConfigurationException(e);
  }
}
Example #2
Source File: ConnectorJmxReporter.java    From mirus with BSD 3-Clause "New" or "Revised" License 6 votes vote down vote up 
public void handleConnector(Herder herder, ConnectorInfo connector) {

    ensureMetricsCreated(connector.name());
    Map<String, Long> stateCounts =
        connector
            .tasks()
            .stream()
            .map(t -> herder.taskStatus(t).state())
            .collect(Collectors.groupingBy(Function.identity(), Collectors.counting()));

    allStates.keySet().forEach(state -> stateCounts.putIfAbsent(state, 0L));

    stateCounts.forEach(
        (state, count) ->
            metrics
                .sensor(calculateSensorName(state.toLowerCase(), connector.name()))
                .record(count, Time.SYSTEM.milliseconds()));
  }
Example #3
Source File: KafkaConnectRunner.java    From camel-kafka-connector with Apache License 2.0 5 votes vote down vote up 
public void initializeConnector(ConnectorPropertyFactory connectorPropertyFactory,
                                Consumer<ConnectorInitState> callback) throws ExecutionException, InterruptedException {
    Properties connectorProps = connectorPropertyFactory.getProperties();

    FutureCallback<Herder.Created<ConnectorInfo>> cb = new FutureCallback<>((error, info) ->
            callback.accept(new ConnectorInitState(info.result().config(), info.created(), error)));

    herder.putConnectorConfig(
            connectorProps.getProperty(ConnectorConfig.NAME_CONFIG),
            Utils.propsToStringMap(connectorProps), false, cb);

    cb.get();
}
Example #4
Source File: KafkaConnectRunner.java    From camel-kafka-connector with Apache License 2.0 5 votes vote down vote up 
public <T> void initializeConnector(ConnectorPropertyFactory connectorPropertyFactory,
                                    BiConsumer<ConnectorInitState, T> callback, T payload) throws ExecutionException, InterruptedException {
    Properties connectorProps = connectorPropertyFactory.getProperties();

    FutureCallback<Herder.Created<ConnectorInfo>> cb = new FutureCallback<>((error, info) ->
            callback.accept(new ConnectorInitState(info.result().config(), info.created(), error), payload));

    herder.putConnectorConfig(
            connectorProps.getProperty(ConnectorConfig.NAME_CONFIG),
            Utils.propsToStringMap(connectorProps), false, cb);

    cb.get();
}
Example #5
Source File: HerderStatusMonitor.java    From mirus with BSD 3-Clause "New" or "Revised" License 5 votes vote down vote up 
HerderStatusMonitor(
    Herder herder,
    String workerId,
    long pollingIntervalMillis,
    boolean autoRestartTaskEnabled,
    boolean autoRestartConnectorEnabled) {
  this.herder = herder;
  this.workerId = workerId;
  this.pollingIntervalMillis = pollingIntervalMillis;
  this.autoRestartTaskEnabled = autoRestartTaskEnabled;
  this.autoRestartConnectorEnabled = autoRestartConnectorEnabled;
  this.shutdownHook = new ShutdownHook();
  logger.info("Task monitor thread will poll tasks every {} ms", pollingIntervalMillis);
}