org.datavec.nlp.tokenization.tokenizerfactory.DefaultTokenizerFactory Java Examples

The following examples show how to use org.datavec.nlp.tokenization.tokenizerfactory.DefaultTokenizerFactory. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: AbstractTfidfVectorizer.java From deeplearning4j with Apache License 2.0

6 votes

@Override
public TokenizerFactory createTokenizerFactory(Configuration conf) {
    String clazz = conf.get(TOKENIZER, DefaultTokenizerFactory.class.getName());
    try {
        Class<? extends TokenizerFactory> tokenizerFactoryClazz =
                        (Class<? extends TokenizerFactory>) Class.forName(clazz);
        TokenizerFactory tf = tokenizerFactoryClazz.newInstance();
        String preproc = conf.get(PREPROCESSOR, null);
        if(preproc != null){
            TokenPreProcess tpp = (TokenPreProcess) Class.forName(preproc).newInstance();
            tf.setTokenPreProcessor(tpp);
        }
        return tf;
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}

Example #2

Source File: VasttextTextVectorizer.java From scava with Eclipse Public License 2.0

5 votes

@Override
public void initialize(Configuration conf) {

	tokenizerFactory = new DefaultTokenizerFactory();
       minWordFrequency = conf.getInt(MIN_WORD_FREQUENCY, 5);
       maxNgrams = conf.getInt(NGRAMS, 1);
       maxSkipBigrams = conf.getInt(SKIP_NGRAMS, 0);

       if(conf.getBoolean(DELETE_STOP_WORDS, false))
       {
       	System.err.println("StopWord Removal: Yes");
        stopWords = conf.getStringCollection(STOP_WORDS);
        if (stopWords == null || stopWords.isEmpty())
            stopWords = StopWords.getStopWords();
       }
       else
       	System.err.println("StopWord Removal: No");
       System.err.println("Freq min:"+minWordFrequency);
       
       System.err.println("N-grams:"+maxNgrams);
       
       System.err.println("Skip bigrams:"+maxSkipBigrams);
       
	cache = new VasttextDictionary();
       cache.initialize(conf);
	cache.setMaxNgrams(maxNgrams);
	cache.setMaxSkipBigrams(maxSkipBigrams);
}

Example #3

Source File: VasttextTextVectorizer.java From scava with Eclipse Public License 2.0

5 votes

public void loadDictionary(Object dictionary) throws FileNotFoundException, IOException, ClassNotFoundException
{
	cache = (VasttextDictionary) dictionary;
	tokenizerFactory = new DefaultTokenizerFactory();
	maxNgrams=cache.getMaxNgrams();
	maxSkipBigrams=cache.getMaxSkipBigrams();
	featuresStartAt=cache.vocabWords().size();
	setfitFinished();
}

Example #4

Source File: AbstractTfidfVectorizer.java From DataVec with Apache License 2.0

5 votes

@Override
public TokenizerFactory createTokenizerFactory(Configuration conf) {
    String clazz = conf.get(TOKENIZER, DefaultTokenizerFactory.class.getName());
    try {
        Class<? extends TokenizerFactory> tokenizerFactoryClazz =
                        (Class<? extends TokenizerFactory>) Class.forName(clazz);
        return tokenizerFactoryClazz.newInstance();
    } catch (Exception e) {
        throw new RuntimeException(e);
    }
}