org.eclipse.xtext.parser.antlr.TokenTool Java Examples

The following examples show how to use org.eclipse.xtext.parser.antlr.TokenTool. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: TokenSequencePreservingPartialParsingHelper.java    From xtext-extras with Eclipse Public License 2.0 5 votes vote down vote up 
protected boolean isSameTokenSequence(TokenSource originalSource, TokenSource newSource, int expectedLength) {
	Token token = originalSource.nextToken();
	int newLength = 0;
	while(Token.EOF != token.getType()) {
		Token newToken = newSource.nextToken();
		if (token.getType() != newToken.getType()) {
			return false;
		}
		newLength += TokenTool.getLength(newToken);
		token = originalSource.nextToken();
	}
	return newLength == expectedLength;
}
Example #2
Source File: TemplateBodyHighlighter.java    From xtext-eclipse with Eclipse Public License 2.0 5 votes vote down vote up 
protected void doProvideHighlightingFor(String body, org.eclipse.xtext.ide.editor.syntaxcoloring.IHighlightedPositionAcceptor acceptor) {
	lexer.setCharStream(new ANTLRStringStream(body));
	Token token = lexer.nextToken();
	while(token != Token.EOF_TOKEN) {
		String id = tokenIdMapper.getId(token.getType());
		int offset = TokenTool.getOffset(token);
		int length = TokenTool.getLength(token);
		acceptor.addPosition(offset, length, id);
		token = lexer.nextToken();
	}
}
Example #3
Source File: FlexerBasedTemplateBodyHighlighter.java    From xtext-xtend with Eclipse Public License 2.0 5 votes vote down vote up 
@Override
public void doProvideHighlightingFor(final String body, final IHighlightedPositionAcceptor acceptor) {
  StringReader _stringReader = new StringReader(body);
  final FlexTokenSource tokenSource = this._flexerFactory.createTokenSource(_stringReader);
  Token token = tokenSource.nextToken();
  while ((!Objects.equal(token, Token.EOF_TOKEN))) {
    {
      final String id = this._abstractAntlrTokenToAttributeIdMapper.getId(token.getType());
      final int offset = TokenTool.getOffset(token);
      final int length = TokenTool.getLength(token);
      acceptor.addPosition(offset, length, id);
      token = tokenSource.nextToken();
    }
  }
}
Example #4
Source File: AbstractLexerBasedConverter.java    From xtext-core with Eclipse Public License 2.0 4 votes vote down vote up 
protected String getRuleName(Token token) {
	String result = getTokenDefMap().get(token.getType());
	return TokenTool.getLexerRuleName(result);
}