Java Code Examples for org.apache.hadoop.conf.Configuration#getPropertySources()

The following examples show how to use org.apache.hadoop.conf.Configuration#getPropertySources() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: JobHistoryUtils.java    From hadoop with Apache License 2.0 5 votes vote down vote up 
/**
 * Get default file system URI for the cluster (used to ensure consistency
 * of history done/staging locations) over different context
 *
 * @return Default file context
 */
private static FileContext getDefaultFileContext() {
  // If FS_DEFAULT_NAME_KEY was set solely by core-default.xml then we ignore
  // ignore it. This prevents defaulting history paths to file system specified
  // by core-default.xml which would not make sense in any case. For a test
  // case to exploit this functionality it should create core-site.xml
  FileContext fc = null;
  Configuration defaultConf = new Configuration();
  String[] sources;
  sources = defaultConf.getPropertySources(
      CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY);
  if (sources != null &&
      (!Arrays.asList(sources).contains("core-default.xml") ||
      sources.length > 1)) {
    try {
      fc = FileContext.getFileContext(defaultConf);
      LOG.info("Default file system [" +
                fc.getDefaultFileSystem().getUri() + "]");
    } catch (UnsupportedFileSystemException e) {
      LOG.error("Unable to create default file context [" +
          defaultConf.get(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY) +
          "]",
          e);
    }
  }
  else {
    LOG.info("Default file system is set solely " +
        "by core-default.xml therefore -  ignoring");
  }

  return fc;
}
Example 2
Source File: ApplicationMasterMain.java    From twill with Apache License 2.0 5 votes vote down vote up 
/**
 * Optionally sets the RM scheduler address based on the environment variable if it is not set in the cluster config.
 */
private static void setRMSchedulerAddress(Configuration conf, String schedulerAddress) {
  if (schedulerAddress == null) {
    return;
  }

  // If the RM scheduler address is not in the config or it's from yarn-default.xml,
  // replace it with the one from the env, which is the same as the one client connected to.
  String[] sources = conf.getPropertySources(YarnConfiguration.RM_SCHEDULER_ADDRESS);
  if (sources == null || sources.length == 0 || "yarn-default.xml".equals(sources[sources.length - 1])) {
    conf.set(YarnConfiguration.RM_SCHEDULER_ADDRESS, schedulerAddress);
  }
}
Example 3
Source File: JobHistoryUtils.java    From big-c with Apache License 2.0 5 votes vote down vote up 
/**
 * Get default file system URI for the cluster (used to ensure consistency
 * of history done/staging locations) over different context
 *
 * @return Default file context
 */
private static FileContext getDefaultFileContext() {
  // If FS_DEFAULT_NAME_KEY was set solely by core-default.xml then we ignore
  // ignore it. This prevents defaulting history paths to file system specified
  // by core-default.xml which would not make sense in any case. For a test
  // case to exploit this functionality it should create core-site.xml
  FileContext fc = null;
  Configuration defaultConf = new Configuration();
  String[] sources;
  sources = defaultConf.getPropertySources(
      CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY);
  if (sources != null &&
      (!Arrays.asList(sources).contains("core-default.xml") ||
      sources.length > 1)) {
    try {
      fc = FileContext.getFileContext(defaultConf);
      LOG.info("Default file system [" +
                fc.getDefaultFileSystem().getUri() + "]");
    } catch (UnsupportedFileSystemException e) {
      LOG.error("Unable to create default file context [" +
          defaultConf.get(CommonConfigurationKeysPublic.FS_DEFAULT_NAME_KEY) +
          "]",
          e);
    }
  }
  else {
    LOG.info("Default file system is set solely " +
        "by core-default.xml therefore -  ignoring");
  }

  return fc;
}
Example 4
Source File: HadoopUtils.java    From mr4c with Apache License 2.0 5 votes vote down vote up 
/**
  * Generates human readable string with property name, value, and source
*/
public static String describeConfProp(Configuration conf, String name) {
	String val = conf.get(name);
	String[] srcs = conf.getPropertySources(name);
	String source = srcs==null ? "unknown" : Arrays.toString(srcs);
	return String.format("%s=%s; source: %s", name, val, source);
}