com.sun.tools.javac.parser.JavaTokenizer Java Examples

The following examples show how to use com.sun.tools.javac.parser.JavaTokenizer. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.

Example #1

Source File: JavaLexerTest.java From openjdk-jdk9 with GNU General Public License v2.0

6 votes

void run() throws Exception {
    Context ctx = new Context();
    Log log = Log.instance(ctx);
    String input = "0bL 0b20L 0xL ";
    log.useSource(new SimpleJavaFileObject(new URI("mem://Test.java"), JavaFileObject.Kind.SOURCE) {
        @Override
        public CharSequence getCharContent(boolean ignoreEncodingErrors) throws IOException {
            return input;
        }
    });
    char[] inputArr = input.toCharArray();
    JavaTokenizer tokenizer = new JavaTokenizer(ScannerFactory.instance(ctx), inputArr, inputArr.length) {
    };

    assertKind(input, tokenizer, TokenKind.LONGLITERAL, "0bL");
    assertKind(input, tokenizer, TokenKind.LONGLITERAL, "0b20L");
    assertKind(input, tokenizer, TokenKind.LONGLITERAL, "0xL");
}

Example #2

Source File: JavaLexerTest.java From openjdk-jdk9 with GNU General Public License v2.0

5 votes

void assertKind(String input, JavaTokenizer tokenizer, TokenKind kind, String expectedText) {
    Token token = tokenizer.readToken();

    if (token.kind != kind) {
        throw new AssertionError("Unexpected token kind: " + token.kind);
    }

    String actualText = input.substring(token.pos, token.endPos);

    if (!Objects.equals(actualText, expectedText)) {
        throw new AssertionError("Unexpected token text: " + actualText);
    }
}

Example #3

Source File: ManParserFactory.java From manifold with Apache License 2.0

4 votes

ManScanner( ManScannerFactory manScannerFactory, JavaTokenizer manJavadocTokenizer )
{
  super( manScannerFactory, manJavadocTokenizer );
}

Example #4

Source File: JavacTokens.java From google-java-format with Apache License 2.0

4 votes

protected AccessibleScanner(ScannerFactory fac, JavaTokenizer tokenizer) {
  super(fac, tokenizer);
}