Package weka.core.tokenizers
-
Class Summary Class Description AlphabeticTokenizer Alphabetic string tokenizer, tokens are to be formed only from contiguous alphabetic sequences.CharacterDelimitedTokenizer Abstract superclass for tokenizers that take characters as delimiters.NGramTokenizer Splits a string into an n-gram with min and max grams.Tokenizer A superclass for all tokenizer algorithms.WordTokenizer A simple tokenizer that is using the java.util.StringTokenizer class to tokenize the strings.