Uses of Interface
edu.illinois.cs.cogcomp.nlp.tokenizer.Tokenizer

Packages that use Tokenizer

Package	Description
edu.illinois.cs.cogcomp.nlp.tokenizer
edu.illinois.cs.cogcomp.nlp.utility
edu.illinois.cs.cogcomp.tokenizer

Uses of Tokenizer in edu.illinois.cs.cogcomp.nlp.tokenizer

Classes in edu.illinois.cs.cogcomp.nlp.tokenizer that implement Tokenizer

Modifier and Type	Class and Description
class	IllinoisTokenizer Deprecated. use StatefulTokenizer instead.
class	StatefulTokenizer This is the entry point to the tokenizer state machine.

Uses of Tokenizer in edu.illinois.cs.cogcomp.nlp.utility

Methods in edu.illinois.cs.cogcomp.nlp.utility with parameters of type Tokenizer

Modifier and Type	Method and Description
static SpanLabelView	TokenizerUtilities.addTokenView(TextAnnotation input, Tokenizer tokenizer, String source)

Constructors in edu.illinois.cs.cogcomp.nlp.utility with parameters of type Tokenizer

Constructor and Description
TokenizerTextAnnotationBuilder(Tokenizer tokenizer) instantiate a TextAnnotationBuilder.

Uses of Tokenizer in edu.illinois.cs.cogcomp.tokenizer

Classes in edu.illinois.cs.cogcomp.tokenizer that implement Tokenizer

Modifier and Type	Class and Description
class	CharacterTokenizer Created by ctsai12 on 11/29/15.
class	ChineseTokenizer Created by ctsai12 on 12/7/15.
class	EnglishTokenizer Created by ctsai12 on 12/13/15.
class	StanfordAnalyzer Created by ctsai12 on 11/21/15.
class	ThaiTokenizer Created by ctsai12 on 3/24/16.
class	WhiteSpaceTokenizer Created by ctsai12 on 11/29/15.