Java Code Examples for org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord#getRecord()

The following examples show how to use org.apache.beam.sdk.io.gcp.bigquery.SchemaAndRecord#getRecord() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: BeamUtils.java    From nomulus with Apache License 2.0 6 votes vote down vote up 
/**
 * Checks that no expected fields in the record are missing.
 *
 * <p>Note that this simply makes sure the field is not null; it may still generate a parse error
 * when interpreting the string representation of an object.
 *
 * @throws IllegalStateException if the record returns null for any field in {@code fieldNames}
 */
public static void checkFieldsNotNull(
    ImmutableList<String> fieldNames, SchemaAndRecord schemaAndRecord) {
  GenericRecord record = schemaAndRecord.getRecord();
  ImmutableList<String> nullFields =
      fieldNames
          .stream()
          .filter(fieldName -> record.get(fieldName) == null)
          .collect(ImmutableList.toImmutableList());
  String missingFieldList = Joiner.on(", ").join(nullFields);
  if (!nullFields.isEmpty()) {
    throw new IllegalStateException(
        String.format(
            "Read unexpected null value for field(s) %s for record %s",
            missingFieldList, record));
  }
}
Example 2
Source File: BigQueryToTFRecord.java    From DataflowTemplates with Apache License 2.0 5 votes vote down vote up 
/**
 * The {@link BigQueryToTFRecord#record2Example(SchemaAndRecord)} method uses takes in a
 * SchemaAndRecord Object returned from a BigQueryIO.read() step and builds a TensorFlow Example
 * from the record.
 */
@VisibleForTesting
protected static byte[] record2Example(SchemaAndRecord schemaAndRecord) {
  Example.Builder example = Example.newBuilder();
  Features.Builder features = example.getFeaturesBuilder();
  GenericRecord record = schemaAndRecord.getRecord();
  for (TableFieldSchema field : schemaAndRecord.getTableSchema().getFields()) {
    Feature feature = buildFeature(record.get(field.getName()), field.getType());
    features.putFeature(field.getName(), feature);
  }
  return example.build().toByteArray();
}
Example 3
Source File: BillingEvent.java    From nomulus with Apache License 2.0 5 votes vote down vote up 
/**
 * Constructs a {@code BillingEvent} from a {@code SchemaAndRecord}.
 *
 * @see <a
 *     href=http://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/generic/GenericData.Record.html>
 *     Apache AVRO GenericRecord</a>
 */
static BillingEvent parseFromRecord(SchemaAndRecord schemaAndRecord) {
  checkFieldsNotNull(FIELD_NAMES, schemaAndRecord);
  GenericRecord record = schemaAndRecord.getRecord();
  String flags = extractField(record, "flags");
  double amount = getDiscountedAmount(Double.parseDouble(extractField(record, "amount")), flags);
  return create(
      // We need to chain parsers off extractField because GenericRecord only returns
      // Objects, which contain a string representation of their underlying types.
      Long.parseLong(extractField(record, "id")),
      // Bigquery provides UNIX timestamps with microsecond precision.
      Instant.ofEpochMilli(Long.parseLong(extractField(record, "billingTime")) / 1000)
          .atZone(ZoneId.of("UTC")),
      Instant.ofEpochMilli(Long.parseLong(extractField(record, "eventTime")) / 1000)
          .atZone(ZoneId.of("UTC")),
      extractField(record, "registrarId"),
      extractField(record, "billingId"),
      extractField(record, "poNumber"),
      extractField(record, "tld"),
      extractField(record, "action"),
      extractField(record, "domain"),
      extractField(record, "repositoryId"),
      Integer.parseInt(extractField(record, "years")),
      extractField(record, "currency"),
      amount,
      flags);
}
Example 4
Source File: Subdomain.java    From nomulus with Apache License 2.0 5 votes vote down vote up 
/**
 * Constructs a {@link Subdomain} from an Apache Avro {@code SchemaAndRecord}.
 *
 * @see <a
 *     href=http://avro.apache.org/docs/1.7.7/api/java/org/apache/avro/generic/GenericData.Record.html>
 *     Apache AVRO GenericRecord</a>
 */
static Subdomain parseFromRecord(SchemaAndRecord schemaAndRecord) {
  checkFieldsNotNull(FIELD_NAMES, schemaAndRecord);
  GenericRecord record = schemaAndRecord.getRecord();
  return create(
      extractField(record, "fullyQualifiedDomainName"),
      extractField(record, "registrarClientId"),
      extractField(record, "registrarEmailAddress"));
}