Java Code Examples for org.apache.hadoop.hive.ql.metadata.Partition#getDataLocation()

The following examples show how to use org.apache.hadoop.hive.ql.metadata.Partition#getDataLocation() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: HdfsModifiedTimeHiveVersionFinder.java    From incubator-gobblin with Apache License 2.0 6 votes vote down vote up 
/**
 * Create a {@link TimestampedHiveDatasetVersion} from a {@link Partition} based on the Modified time of underlying
 * hdfs data location
 * @throws IllegalArgumentException when argument is null
 * @throws IllegalArgumentException when data location of partition is null
 * @throws IllegalArgumentException when data location of partition doesn't exist
 * {@inheritDoc}
 */
@Override
protected TimestampedHiveDatasetVersion getDatasetVersion(Partition partition) {
  try {
    Preconditions.checkArgument(partition != null, "Argument to method ");

    Path dataLocation = partition.getDataLocation();
    Preconditions
        .checkArgument(dataLocation != null, "Data location is null for partition " + partition.getCompleteName());
    boolean exists = this.fs.exists(dataLocation);
    Preconditions.checkArgument(exists, "Data location doesn't exist for partition " + partition.getCompleteName());

    long modificationTS = this.fs.getFileStatus(dataLocation).getModificationTime();
    return new TimestampedHiveDatasetVersion(new DateTime(modificationTS), partition);
  } catch (IOException e) {
    throw new RuntimeException(e);
  }
}
Example 2
Source File: HiveCopyEntityHelper.java    From incubator-gobblin with Apache License 2.0 5 votes vote down vote up 
int addPartitionDeregisterSteps(List<CopyEntity> copyEntities, String fileSet, int initialPriority,
    Table table, Partition partition) throws IOException {

  int stepPriority = initialPriority;
  Collection<Path> partitionPaths = Lists.newArrayList();

  if (this.deleteMethod == DeregisterFileDeleteMethod.RECURSIVE) {
    partitionPaths = Lists.newArrayList(partition.getDataLocation());
  } else if (this.deleteMethod == DeregisterFileDeleteMethod.INPUT_FORMAT) {
    InputFormat<?, ?> inputFormat = HiveUtils.getInputFormat(partition.getTPartition().getSd());

    HiveLocationDescriptor targetLocation = new HiveLocationDescriptor(partition.getDataLocation(), inputFormat,
        this.targetFs, this.dataset.getProperties());

    partitionPaths = targetLocation.getPaths().keySet();
  } else if (this.deleteMethod == DeregisterFileDeleteMethod.NO_DELETE) {
    partitionPaths = Lists.newArrayList();
  }

  if (!partitionPaths.isEmpty()) {
    DeleteFileCommitStep deletePaths = DeleteFileCommitStep.fromPaths(this.targetFs, partitionPaths,
        this.dataset.getProperties(), table.getDataLocation());
    copyEntities.add(new PostPublishStep(fileSet, Maps.<String, String> newHashMap(), deletePaths, stepPriority++));
  }

  PartitionDeregisterStep deregister =
      new PartitionDeregisterStep(table.getTTable(), partition.getTPartition(), this.targetURI, this.hiveRegProps);
  copyEntities.add(new PostPublishStep(fileSet, Maps.<String, String> newHashMap(), deregister, stepPriority++));
  return stepPriority;
}
Example 3
Source File: TimestampedHiveDatasetVersion.java    From incubator-gobblin with Apache License 2.0 4 votes vote down vote up 
public TimestampedHiveDatasetVersion(DateTime version, Partition partition) {
  super(version, partition.getDataLocation());
  this.partition = partition;
}
Example 4
Source File: HiveLocationDescriptor.java    From incubator-gobblin with Apache License 2.0 4 votes vote down vote up 
public static HiveLocationDescriptor forPartition(Partition partition, FileSystem fs, Properties properties) throws IOException {
  return new HiveLocationDescriptor(partition.getDataLocation(),
      HiveUtils.getInputFormat(partition.getTPartition().getSd()), fs, properties);
}