Python tokenize Module

This page shows the popular functions and classes defined in the tokenize module. The items are ordered by their popularity in 40,000 open source Python projects. If you can not find a good example below, you can try the search function to search modules.

• 1. generate_tokens()    
  Used in 144 projects
• 2. COMMENT    
  Used in 101 projects
• 3. TokenError()    
  Used in 89 projects
• 4. NAME    
  Used in 84 projects
• 5. detect_encoding()    
  Used in 72 projects
• 6. NEWLINE    
  Used in 71 projects
• 7. tokenize()    
  Used in 69 projects
• 8. NL    
  Used in 67 projects
• 9. INDENT    
  Used in 57 projects
• 10. STRING    
  Used in 56 projects
• 11. DEDENT    
  Used in 50 projects
• 12. OP    
  Used in 46 projects
• 13. open()    
  Used in 44 projects
• 14. NUMBER    
  Used in 41 projects
• 15. untokenize()    
  Used in 40 projects
• 16. ENDMARKER    
  Used in 20 projects
• 17. tok_name()    
  Used in 15 projects
• 18. ENCODING    
  Used in 14 projects
• 19. Name()    
  Used in 14 projects
• 20. ERRORTOKEN    
  Used in 13 projects
• 21. TokenInfo()    
  Used in 10 projects