Java Code Examples for org.apache.hadoop.util.StringUtils#escapeString()

The following examples show how to use org.apache.hadoop.util.StringUtils#escapeString() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: FileInputFormat.java    From hadoop with Apache License 2.0 5 votes vote down vote up 
/**
 * Set the array of {@link Path}s as the list of inputs
 * for the map-reduce job.
 * 
 * @param job The job to modify 
 * @param inputPaths the {@link Path}s of the input directories/files 
 * for the map-reduce job.
 */ 
public static void setInputPaths(Job job, 
                                 Path... inputPaths) throws IOException {
  Configuration conf = job.getConfiguration();
  Path path = inputPaths[0].getFileSystem(conf).makeQualified(inputPaths[0]);
  StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
  for(int i = 1; i < inputPaths.length;i++) {
    str.append(StringUtils.COMMA_STR);
    path = inputPaths[i].getFileSystem(conf).makeQualified(inputPaths[i]);
    str.append(StringUtils.escapeString(path.toString()));
  }
  conf.set(INPUT_DIR, str.toString());
}
Example 2
Source File: FileInputFormat.java    From hadoop-gpu with Apache License 2.0 5 votes vote down vote up 
/**
 * Add a {@link Path} to the list of inputs for the map-reduce job.
 * 
 * @param job The {@link Job} to modify
 * @param path {@link Path} to be added to the list of inputs for 
 *            the map-reduce job.
 */
public static void addInputPath(Job job, 
                                Path path) throws IOException {
  Configuration conf = job.getConfiguration();
  FileSystem fs = FileSystem.get(conf);
  path = path.makeQualified(fs);
  String dirStr = StringUtils.escapeString(path.toString());
  String dirs = conf.get("mapred.input.dir");
  conf.set("mapred.input.dir", dirs == null ? dirStr : dirs + "," + dirStr);
}
Example 3
Source File: FileInputFormat.java    From hadoop-gpu with Apache License 2.0 5 votes vote down vote up 
/**
 * Set the array of {@link Path}s as the list of inputs
 * for the map-reduce job.
 * 
 * @param job The job to modify 
 * @param inputPaths the {@link Path}s of the input directories/files 
 * for the map-reduce job.
 */ 
public static void setInputPaths(Job job, 
                                 Path... inputPaths) throws IOException {
  Configuration conf = job.getConfiguration();
  FileSystem fs = FileSystem.get(conf);
  Path path = inputPaths[0].makeQualified(fs);
  StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
  for(int i = 1; i < inputPaths.length;i++) {
    str.append(StringUtils.COMMA_STR);
    path = inputPaths[i].makeQualified(fs);
    str.append(StringUtils.escapeString(path.toString()));
  }
  conf.set("mapred.input.dir", str.toString());
}
Example 4
Source File: JobHistory.java    From hadoop-gpu with Apache License 2.0 5 votes vote down vote up 
/**
 * Parses history file and invokes Listener.handle() for 
 * each line of history. It can be used for looking through history
 * files for specific items without having to keep whole history in memory. 
 * @param path path to history file
 * @param l Listener for history events 
 * @param fs FileSystem where history file is present
 * @throws IOException
 */
public static void parseHistoryFromFS(String path, Listener l, FileSystem fs)
throws IOException{
  FSDataInputStream in = fs.open(new Path(path));
  BufferedReader reader = new BufferedReader(new InputStreamReader (in));
  try {
    String line = null; 
    StringBuffer buf = new StringBuffer(); 
    
    // Read the meta-info line. Note that this might a jobinfo line for files
    // written with older format
    line = reader.readLine();
    
    // Check if the file is empty
    if (line == null) {
      return;
    }
    
    // Get the information required for further processing
    MetaInfoManager mgr = new MetaInfoManager(line);
    boolean isEscaped = mgr.isValueEscaped();
    String lineDelim = String.valueOf(mgr.getLineDelim());  
    String escapedLineDelim = 
      StringUtils.escapeString(lineDelim, StringUtils.ESCAPE_CHAR, 
                               mgr.getLineDelim());
    
    do {
      buf.append(line); 
      if (!line.trim().endsWith(lineDelim) 
          || line.trim().endsWith(escapedLineDelim)) {
        buf.append("\n");
        continue; 
      }
      parseLine(buf.toString(), l, isEscaped);
      buf = new StringBuffer(); 
    } while ((line = reader.readLine())!= null);
  } finally {
    try { reader.close(); } catch (IOException ex) {}
  }
}
Example 5
Source File: FileInputFormat.java    From RDFS with Apache License 2.0 5 votes vote down vote up 
/**
 * Add a {@link Path} to the list of inputs for the map-reduce job.
 * 
 * @param job The {@link Job} to modify
 * @param path {@link Path} to be added to the list of inputs for 
 *            the map-reduce job.
 */
public static void addInputPath(Job job, 
                                Path path) throws IOException {
  Configuration conf = job.getConfiguration();
  FileSystem fs = FileSystem.get(conf);
  path = path.makeQualified(fs);
  String dirStr = StringUtils.escapeString(path.toString());
  String dirs = conf.get("mapred.input.dir");
  conf.set("mapred.input.dir", dirs == null ? dirStr : dirs + "," + dirStr);
}
Example 6
Source File: FileInputFormat.java    From RDFS with Apache License 2.0 5 votes vote down vote up 
/**
 * Set the array of {@link Path}s as the list of inputs
 * for the map-reduce job.
 * 
 * @param job The job to modify 
 * @param inputPaths the {@link Path}s of the input directories/files 
 * for the map-reduce job.
 */ 
public static void setInputPaths(Job job, 
                                 Path... inputPaths) throws IOException {
  Configuration conf = job.getConfiguration();
  FileSystem fs = FileSystem.get(conf);
  Path path = inputPaths[0].makeQualified(fs);
  StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
  for(int i = 1; i < inputPaths.length;i++) {
    str.append(StringUtils.COMMA_STR);
    path = inputPaths[i].makeQualified(fs);
    str.append(StringUtils.escapeString(path.toString()));
  }
  conf.set("mapred.input.dir", str.toString());
}
Example 7
Source File: JobHistory.java    From RDFS with Apache License 2.0 5 votes vote down vote up 
/**
 * Parses history file and invokes Listener.handle() for 
 * each line of history. It can be used for looking through history
 * files for specific items without having to keep whole history in memory. 
 * @param path path to history file
 * @param l Listener for history events 
 * @param fs FileSystem where history file is present
 * @throws IOException
 */
public static void parseHistoryFromFS(String path, Listener l, FileSystem fs)
throws IOException{
  FSDataInputStream in = fs.open(new Path(path));
  BufferedReader reader = new BufferedReader(new InputStreamReader (in));
  try {
    String line = null; 
    StringBuffer buf = new StringBuffer(); 
    
    // Read the meta-info line. Note that this might a jobinfo line for files
    // written with older format
    line = reader.readLine();
    
    // Check if the file is empty
    if (line == null) {
      return;
    }
    
    // Get the information required for further processing
    MetaInfoManager mgr = new MetaInfoManager(line);
    boolean isEscaped = mgr.isValueEscaped();
    String lineDelim = String.valueOf(mgr.getLineDelim());  
    String escapedLineDelim = 
      StringUtils.escapeString(lineDelim, StringUtils.ESCAPE_CHAR, 
                               mgr.getLineDelim());
    
    do {
      buf.append(line); 
      if (!line.trim().endsWith(lineDelim) 
          || line.trim().endsWith(escapedLineDelim)) {
        buf.append("\n");
        continue; 
      }
      parseLine(buf.toString(), l, isEscaped);
      buf = new StringBuffer(); 
    } while ((line = reader.readLine())!= null);
  } finally {
    try { reader.close(); } catch (IOException ex) {}
  }
}
Example 8
Source File: FileInputFormat.java    From big-c with Apache License 2.0 5 votes vote down vote up 
/**
 * Add a {@link Path} to the list of inputs for the map-reduce job.
 * 
 * @param job The {@link Job} to modify
 * @param path {@link Path} to be added to the list of inputs for 
 *            the map-reduce job.
 */
public static void addInputPath(Job job, 
                                Path path) throws IOException {
  Configuration conf = job.getConfiguration();
  path = path.getFileSystem(conf).makeQualified(path);
  String dirStr = StringUtils.escapeString(path.toString());
  String dirs = conf.get(INPUT_DIR);
  conf.set(INPUT_DIR, dirs == null ? dirStr : dirs + "," + dirStr);
}
Example 9
Source File: FileInputFormat.java    From big-c with Apache License 2.0 5 votes vote down vote up 
/**
 * Set the array of {@link Path}s as the list of inputs
 * for the map-reduce job.
 * 
 * @param job The job to modify 
 * @param inputPaths the {@link Path}s of the input directories/files 
 * for the map-reduce job.
 */ 
public static void setInputPaths(Job job, 
                                 Path... inputPaths) throws IOException {
  Configuration conf = job.getConfiguration();
  Path path = inputPaths[0].getFileSystem(conf).makeQualified(inputPaths[0]);
  StringBuffer str = new StringBuffer(StringUtils.escapeString(path.toString()));
  for(int i = 1; i < inputPaths.length;i++) {
    str.append(StringUtils.COMMA_STR);
    path = inputPaths[i].getFileSystem(conf).makeQualified(inputPaths[i]);
    str.append(StringUtils.escapeString(path.toString()));
  }
  conf.set(INPUT_DIR, str.toString());
}
Example 10
Source File: FileInputFormat.java    From hadoop with Apache License 2.0 5 votes vote down vote up 
/**
 * Add a {@link Path} to the list of inputs for the map-reduce job.
 * 
 * @param job The {@link Job} to modify
 * @param path {@link Path} to be added to the list of inputs for 
 *            the map-reduce job.
 */
public static void addInputPath(Job job, 
                                Path path) throws IOException {
  Configuration conf = job.getConfiguration();
  path = path.getFileSystem(conf).makeQualified(path);
  String dirStr = StringUtils.escapeString(path.toString());
  String dirs = conf.get(INPUT_DIR);
  conf.set(INPUT_DIR, dirs == null ? dirStr : dirs + "," + dirStr);
}
Example 11
Source File: CountersStrings.java    From big-c with Apache License 2.0 4 votes vote down vote up 
private static String escape(String string) {
  return StringUtils.escapeString(string, StringUtils.ESCAPE_CHAR,
                                  charsToEscape);
}
Example 12
Source File: JobHistory.java    From RDFS with Apache License 2.0 4 votes vote down vote up 
/** Escapes the string especially for {@link JobHistory}
 */
public static String escapeString(String data) {
  return StringUtils.escapeString(data, StringUtils.ESCAPE_CHAR, 
                                  charsToEscape);
}
Example 13
Source File: Counters.java    From RDFS with Apache License 2.0 4 votes vote down vote up 
private static String escape(String string) {
  return StringUtils.escapeString(string, StringUtils.ESCAPE_CHAR, 
                                  charsToEscape);
}
Example 14
Source File: JobInProgress.java    From RDFS with Apache License 2.0 4 votes vote down vote up 
/**
 * Log a summary of the job's runtime.
 *
 * @param job {@link JobInProgress} whose summary is to be logged, cannot
 *            be <code>null</code>.
 * @param cluster {@link ClusterStatus} of the cluster on which the job was
 *                run, cannot be <code>null</code>
 */
public static void logJobSummary(JobInProgress job, ClusterStatus cluster) {
  JobStatus status = job.getStatus();
  JobProfile profile = job.getProfile();
  String user = StringUtils.escapeString(profile.getUser(),
                                         StringUtils.ESCAPE_CHAR,
                                         charsToEscape);
  String queue = StringUtils.escapeString(profile.getQueueName(),
                                          StringUtils.ESCAPE_CHAR,
                                          charsToEscape);
  Counters jobCounters = job.getJobCounters();
  long mapSlotSeconds =
    (jobCounters.getCounter(Counter.SLOTS_MILLIS_MAPS) +
     jobCounters.getCounter(Counter.FALLOW_SLOTS_MILLIS_MAPS)) / 1000;
  long reduceSlotSeconds =
    (jobCounters.getCounter(Counter.SLOTS_MILLIS_REDUCES) +
     jobCounters.getCounter(Counter.FALLOW_SLOTS_MILLIS_REDUCES)) / 1000;

  LOG.info("jobId=" + job.getJobID() + StringUtils.COMMA +
           "submitTime" + EQUALS + job.getStartTime() + StringUtils.COMMA +
           "launchTime" + EQUALS + job.getLaunchTime() + StringUtils.COMMA +
           "finishTime" + EQUALS + job.getFinishTime() + StringUtils.COMMA +
           "numMaps" + EQUALS + job.getTasks(TaskType.MAP).length +
                       StringUtils.COMMA +
           "numSlotsPerMap" + EQUALS + job.getNumSlotsPerMap() +
                              StringUtils.COMMA +
           "numReduces" + EQUALS + job.getTasks(TaskType.REDUCE).length +
                          StringUtils.COMMA +
           "numSlotsPerReduce" + EQUALS + job.getNumSlotsPerReduce() +
                                 StringUtils.COMMA +
           "user" + EQUALS + user + StringUtils.COMMA +
           "queue" + EQUALS + queue + StringUtils.COMMA +
           "status" + EQUALS +
                      JobStatus.getJobRunState(status.getRunState()) +
                      StringUtils.COMMA +
           "mapSlotSeconds" + EQUALS + mapSlotSeconds + StringUtils.COMMA +
           "reduceSlotsSeconds" + EQUALS + reduceSlotSeconds  +
                                  StringUtils.COMMA +
           "clusterMapCapacity" + EQUALS + cluster.getMaxMapTasks() +
                                  StringUtils.COMMA +
           "clusterReduceCapacity" + EQUALS + cluster.getMaxReduceTasks()
  );
}
Example 15
Source File: JobImpl.java    From big-c with Apache License 2.0 4 votes vote down vote up 
public static String escapeString(String data) {
  return StringUtils.escapeString(data, StringUtils.ESCAPE_CHAR,
      new char[] {'"', '=', '.'});
}
Example 16
Source File: CoronaJobInProgress.java    From RDFS with Apache License 2.0 4 votes vote down vote up 
/**
 * Log a summary of the job's runtime.
 *
 * @param job {@link JobInProgress} whose summary is to be logged, cannot
 *            be <code>null</code>.
 */
@SuppressWarnings("deprecation")
public static void logJobSummary(CoronaJobInProgress job) {
  JobStatus status = job.getStatus();
  JobProfile profile = job.getProfile();
  String user = StringUtils.escapeString(profile.getUser(),
                                         StringUtils.ESCAPE_CHAR,
                                         charsToEscape);
  String queue = StringUtils.escapeString(profile.getQueueName(),
                                          StringUtils.ESCAPE_CHAR,
                                          charsToEscape);
  Counters jobCounters = job.getJobCounters();
  long mapSlotSeconds =
    (jobCounters.getCounter(Counter.SLOTS_MILLIS_MAPS) +
     jobCounters.getCounter(Counter.FALLOW_SLOTS_MILLIS_MAPS)) / 1000;
  long reduceSlotSeconds =
    (jobCounters.getCounter(Counter.SLOTS_MILLIS_REDUCES) +
     jobCounters.getCounter(Counter.FALLOW_SLOTS_MILLIS_REDUCES)) / 1000;

  LOG.info("jobId=" + profile.getJobID() + StringUtils.COMMA +
           "submitTime" + EQUALS + job.getStartTime() + StringUtils.COMMA +
           "launchTime" + EQUALS + job.getLaunchTime() + StringUtils.COMMA +
           "finishTime" + EQUALS + job.getFinishTime() + StringUtils.COMMA +
           "numMaps" + EQUALS + job.getTasks(TaskType.MAP).length +
                       StringUtils.COMMA +
           "numSlotsPerMap" + EQUALS + NUM_SLOTS_PER_MAP +
                              StringUtils.COMMA +
           "numReduces" + EQUALS + job.getTasks(TaskType.REDUCE).length +
                          StringUtils.COMMA +
           "numSlotsPerReduce" + EQUALS + NUM_SLOTS_PER_REDUCE +
                                 StringUtils.COMMA +
           "user" + EQUALS + user + StringUtils.COMMA +
           "queue" + EQUALS + queue + StringUtils.COMMA +
           "status" + EQUALS +
                      JobStatus.getJobRunState(status.getRunState()) +
                      StringUtils.COMMA +
           "mapSlotSeconds" + EQUALS + mapSlotSeconds + StringUtils.COMMA +
           "reduceSlotsSeconds" + EQUALS + reduceSlotSeconds  +
                                  StringUtils.COMMA
  );
}
Example 17
Source File: JobHistory.java    From hadoop-gpu with Apache License 2.0 4 votes vote down vote up 
/** Escapes the string especially for {@link JobHistory}
 */
static String escapeString(String data) {
  return StringUtils.escapeString(data, StringUtils.ESCAPE_CHAR, 
                                  charsToEscape);
}
Example 18
Source File: CountersStrings.java    From hadoop with Apache License 2.0 4 votes vote down vote up 
private static String escape(String string) {
  return StringUtils.escapeString(string, StringUtils.ESCAPE_CHAR,
                                  charsToEscape);
}
Example 19
Source File: Counters.java    From hadoop-gpu with Apache License 2.0 4 votes vote down vote up 
private static String escape(String string) {
  return StringUtils.escapeString(string, StringUtils.ESCAPE_CHAR, 
                                  charsToEscape);
}
Example 20
Source File: JobImpl.java    From hadoop with Apache License 2.0 4 votes vote down vote up 
public static String escapeString(String data) {
  return StringUtils.escapeString(data, StringUtils.ESCAPE_CHAR,
      new char[] {'"', '=', '.'});
}