org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter Java Examples

The following examples show how to use org.apache.lucene.analysis.miscellaneous.StemmerOverrideFilter. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: StemmerOverrideTokenFilterFactory.java From Elasticsearch with Apache License 2.0

6 votes

static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}

Example #2

Source File: DutchAnalyzer.java From lucene-solr with Apache License 2.0

6 votes

public DutchAnalyzer(CharArraySet stopwords, CharArraySet stemExclusionTable, CharArrayMap<String> stemOverrideDict) {
  this.stoptable = CharArraySet.unmodifiableSet(CharArraySet.copy(stopwords));
  this.excltable = CharArraySet.unmodifiableSet(CharArraySet.copy(stemExclusionTable));
  if (stemOverrideDict.isEmpty()) {
    this.stemdict = null;
  } else {
    // we don't need to ignore case here since we lowercase in this analyzer anyway
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    CharArrayMap<String>.EntryIterator iter = stemOverrideDict.entrySet().iterator();
    CharsRefBuilder spare = new CharsRefBuilder();
    while (iter.hasNext()) {
      char[] nextKey = iter.nextKey();
      spare.copyChars(nextKey, 0, nextKey.length);
      builder.add(spare.get(), iter.currentValue());
    }
    try {
      this.stemdict = builder.build();
    } catch (IOException ex) {
      throw new RuntimeException("can not build stem dict", ex);
    }
  }
}

Example #3

Source File: StemmerOverrideTokenFilterFactory.java From crate with Apache License 2.0

6 votes

static void parseRules(List<String> rules, StemmerOverrideFilter.Builder builder, String mappingSep) {
    for (String rule : rules) {
        String key, override;
        List<String> mapping = Strings.splitSmart(rule, mappingSep, false);
        if (mapping.size() == 2) {
            key = mapping.get(0).trim();
            override = mapping.get(1).trim();
        } else {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        }

        if (key.isEmpty() || override.isEmpty()) {
            throw new RuntimeException("Invalid Keyword override Rule:" + rule);
        } else {
            builder.add(key, override);
        }
    }
}

Example #4

Source File: StemmerOverrideTokenFilterFactory.java From Elasticsearch with Apache License 2.0

5 votes

@Inject
public StemmerOverrideTokenFilterFactory(Index index, IndexSettingsService indexSettingsService, Environment env, @Assisted String name, @Assisted Settings settings) throws IOException {
    super(index, indexSettingsService.getSettings(), name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }
    
    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}

Example #5

Source File: StemmerOverrideTokenFilterFactory.java From crate with Apache License 2.0

5 votes

StemmerOverrideTokenFilterFactory(IndexSettings indexSettings, Environment env, String name, Settings settings) throws IOException {
    super(indexSettings, name, settings);

    List<String> rules = Analysis.getWordList(env, settings, "rules");
    if (rules == null) {
        throw new IllegalArgumentException("stemmer override filter requires either `rules` or `rules_path` to be configured");
    }

    StemmerOverrideFilter.Builder builder = new StemmerOverrideFilter.Builder(false);
    parseRules(rules, builder, "=>");
    overrideMap = builder.build();

}

Example #6

Source File: StemmerOverrideTokenFilterFactory.java From Elasticsearch with Apache License 2.0

4 votes

@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}

Example #7

Source File: StemmerOverrideTokenFilterFactory.java From crate with Apache License 2.0

4 votes

@Override
public TokenStream create(TokenStream tokenStream) {
    return new StemmerOverrideFilter(tokenStream, overrideMap);
}