Java Code Examples for org.apache.avro.generic.GenericData#get()

The following examples show how to use org.apache.avro.generic.GenericData#get() . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: AvroDataStreamParser.java    From datacollector with Apache License 2.0 6 votes vote down vote up 
public AvroDataStreamParser(
    ProtoConfigurableEntity.Context context,
    Schema schema,
    String streamName,
    InputStream inputStream,
    long recordCount,
    int maxObjectLength,
    boolean skipAvroUnionIndexes
) throws IOException {
  this.context = context;
  avroSchema = schema;
  this.streamName = streamName;
  this.recordCount = recordCount;
  datumReader = new GenericDatumReader<>(avroSchema, avroSchema, GenericData.get()); //Reader schema argument is optional
  overrunInputStream = new OverrunInputStream(inputStream, maxObjectLength, true);
  dataFileStream = new DataFileStream<>(overrunInputStream, datumReader);
  seekToOffset();
  this.skipAvroUnionIndexes = skipAvroUnionIndexes;
}
Example 2
Source File: AvroDataFileParser.java    From datacollector with Apache License 2.0 5 votes vote down vote up 
public AvroDataFileParser(ProtoConfigurableEntity.Context context, Schema schema, File file, String readerOffset, int maxObjectLength, boolean skipUnionIndexes)
  throws IOException {
  this.context = context;
  this.file = file;
  this.skipUnionIndexes = skipUnionIndexes;
  DatumReader<GenericRecord> datumReader = new GenericDatumReader<>(schema, schema, GenericData.get());
  sin = new SeekableOverrunFileInputStream(
    new FileInputStream(file), maxObjectLength, true);
  dataFileReader = new DataFileReader<>(sin, datumReader);
  if(readerOffset != null && !readerOffset.isEmpty() && !"0".equals(readerOffset)) {
    String[] split = readerOffset.split(OFFSET_SEPARATOR);
    if(split.length == 3) {
      //split[0] is the file name
      previousSync = Long.parseLong(split[1]);
      recordCount = Long.parseLong(split[2]);
      seekToOffset();
    } else if (split.length == 2) {
      previousSync = Long.parseLong(split[0]);
      recordCount = Long.parseLong(split[1]);
      seekToOffset();
    } else {
      throw new IllegalArgumentException(Utils.format("Invalid offset {}", readerOffset));
    }
  } else {
    recordCount = 0;
    previousSync = dataFileReader.previousSync();
  }
}
Example 3
Source File: AvroJsonReader.java    From parquet-mr with Apache License 2.0 5 votes vote down vote up 
@SuppressWarnings("unchecked")
public AvroJsonReader(InputStream stream, Schema schema) {
  this.stream = stream;
  this.schema = schema;
  this.model = GenericData.get();
  this.iterator = Iterators.transform(AvroJson.parser(stream),
    node -> (E) AvroJson.convertToAvro(model, node, AvroJsonReader.this.schema));
}
Example 4
Source File: DataModelUtil.java    From kite with Apache License 2.0 5 votes vote down vote up 
/**
 * Get the data model for the given type.
 *
 * @param <E> The entity type
 * @param type The Java class of the entity type
 * @return The appropriate data model for the given type
 */
public static <E> GenericData getDataModelForType(Class<E> type) {
  // Need to check if SpecificRecord first because specific records also
  // implement GenericRecord
  if (SpecificRecord.class.isAssignableFrom(type)) {
    return new SpecificData(type.getClassLoader());
  } else if (IndexedRecord.class.isAssignableFrom(type)) {
    return GenericData.get();
  } else {
    return AllowNulls.get();
  }
}
Example 5
Source File: DataDeserializerTest.java    From data-highway with Apache License 2.0 4 votes vote down vote up 
@Before
public void setUp() {
  underTest = new DataDeserializer(schemaLookup, GenericData.get());
}
Example 6
Source File: ExpressionCallTest.java    From depends with MIT License 4 votes vote down vote up 
public GenericRequestor(Protocol protocol, Transceiver transceiver)
  throws IOException {
  this(protocol, transceiver, GenericData.get());
}
Example 7
Source File: GenericRecordBuilder.java    From avro-util with BSD 2-Clause "Simplified" License 4 votes vote down vote up 
public GenericRecordBuilder(Schema schema) {
  super(schema, GenericData.get());
  this.record = new Record(schema);
}
Example 8
Source File: GenericDataSupplier.java    From parquet-mr with Apache License 2.0 4 votes vote down vote up 
@Override
public GenericData get() {
  return GenericData.get();
}
Example 9
Source File: TestAvroDataSupplier.java    From parquet-mr with Apache License 2.0 4 votes vote down vote up 
@Override
public GenericData get() {
  return GenericData.get();
}