Python tokenization Module

This page shows the popular functions and classes defined in the tokenization module. The items are ordered by their popularity in 40,000 open source Python projects. If you can not find a good example below, you can try the search function to search modules.

• 1. FullTokenizer()    
  Used in 61 projects
• 2. printable_text()    
  Used in 52 projects
• 3. convert_to_unicode()    
  Used in 51 projects
• 4. BasicTokenizer()    
  Used in 38 projects
• 5. WordpieceTokenizer()    
  Used in 30 projects
• 6. _is_control()    
  Used in 30 projects
• 7. _is_punctuation()    
  Used in 30 projects
• 8. _is_whitespace()    
  Used in 30 projects
• 9. convert_tokens_to_ids()    
  Used in 29 projects
• 10. whitespace_tokenize()    
  Used in 29 projects
• 11. validate_case_matches_checkpoint()    
  Used in 28 projects