com.mongodb.hadoop.util.MongoConfigUtil Java Examples

The following examples show how to use com.mongodb.hadoop.util.MongoConfigUtil. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: SchoolProficiencyMapper.java From secure-data-service with Apache License 2.0

6 votes

@SuppressWarnings({ "rawtypes", "unchecked" })
@Override
protected void setup(Mapper.Context context) throws IOException, InterruptedException {
    super.setup(context);

    ConfigSections cfg = JobConfiguration.fromHadoopConfiguration(context.getConfiguration());
    MetadataConfig meta = cfg.getMetadata();
    bands = meta.getCutPoints();

    BSONObject obj = MongoConfigUtil.getFields(context.getConfiguration());
    if (obj != null) {
        fields = obj.keySet().toArray(new String[0]);
    } else {
        throw new IllegalArgumentException("Invalid configuration found. Aggregates must "
            + "specify a the hadoop.map.fields property.");
    }
}

Example #2

Source File: MongoAggFormatterTest.java From secure-data-service with Apache License 2.0

6 votes

/**
 * Test method for
 * {@link org.slc.sli.aggregation.mapreduce.io.MongoAggFormatter#getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext)}
 * .
 *
 * @throws Exception
 */
@Test
public void testGetRecordWriter() throws Exception {

    DBCollection mockCollection = Mockito.mock(DBCollection.class);
    UserGroupInformation ugi = Mockito.mock(UserGroupInformation.class);

    PowerMockito.mockStatic(UserGroupInformation.class);

    Mockito.when(UserGroupInformation.getCurrentUser()).thenReturn(ugi);

    TaskAttemptContext c = new MockTaskAttemptContext();
    Configuration config = c.getConfiguration();

    PowerMockito.mockStatic(MongoConfigUtil.class);
    Mockito.when(MongoConfigUtil.getOutputCollection(config)).thenReturn(mockCollection);

    MongoAggFormatter f = new MongoAggFormatter();
    assertTrue(f.getRecordWriter(new MockTaskAttemptContext()) instanceof MongoAggWriter);
}

Example #3

Source File: MongoAggWriterTest.java From secure-data-service with Apache License 2.0

6 votes

public void expect(T expected) throws IOException {
    this.expected = expected;

    mockCollection = Mockito.mock(DBCollection.class);

    ugi = Mockito.mock(UserGroupInformation.class);

    PowerMockito.mockStatic(UserGroupInformation.class);
    Mockito.when(UserGroupInformation.getCurrentUser()).thenReturn(ugi);

    ctx = new MockTaskAttemptContext();
    config = ctx.getConfiguration();

    PowerMockito.mockStatic(MongoConfigUtil.class);
    Mockito.when(MongoConfigUtil.getOutputCollection(config)).thenReturn(mockCollection);

    key = new TenantAndIdEmittableKey("testTenant", "testId");
    key.setTenantId(new Text("Midgar"));
    key.setId(new Text("abcdefg01234567890"));
    writer = new MongoAggWriter(mockCollection, ctx);
}

Example #4

Source File: BulkImportJob.java From zerowing with MIT License

5 votes

private void setupJob() {
  _job.setInputFormatClass(MongoInputFormat.class);
  _job.setMapperClass(BulkImportMapper.class);
  _job.setMapOutputKeyClass(ImmutableBytesWritable.class);
  _job.setMapOutputValueClass(Put.class);

  MongoConfigUtil.setInputURI(getConfiguration(), _mongoURI);
  MongoConfigUtil.setReadSplitsFromSecondary(getConfiguration(), true);
}

Example #5

Source File: MongoAggFormatter.java From secure-data-service with Apache License 2.0

4 votes

@Override
public RecordWriter<EmittableKey, BSONWritable> getRecordWriter(
    org.apache.hadoop.mapreduce.TaskAttemptContext context) {
    Configuration config = context.getConfiguration();
    return new MongoAggWriter(MongoConfigUtil.getOutputCollection(config), context);
}