edu.illinois.cs.cogcomp.nlp.tokenizer

Interface Tokenizer

All Known Implementing Classes:

CharacterTokenizer, ChineseTokenizer, EnglishTokenizer, IllinoisTokenizer, StanfordAnalyzer, StatefulTokenizer, ThaiTokenizer, WhiteSpaceTokenizer

public interface Tokenizer

Specifies behavior for Tokenizers integrated into Cognitive Computation Group NLP software

Author:

Mark Sammons, Vivek Srikumar, Christos Christodoulopoulos

Nested Class Summary

Nested Classes

Modifier and Type	Interface and Description
static class	Tokenizer.Tokenization

Method Summary

Modifier and Type	Method and Description
Pair<String[],IntPair[]>	tokenizeSentence(String sentence) given a sentence, return a set of tokens and their character offsets
Tokenizer.Tokenization	tokenizeTextSpan(String textSpan) given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Method Detail

tokenizeSentence

Pair<String[],IntPair[]> tokenizeSentence(String sentence)

given a sentence, return a set of tokens and their character offsets

Parameters:

sentence - The sentence string

Returns:

A Pair containing the array of tokens and their character offsets

tokenizeTextSpan

Tokenizer.Tokenization tokenizeTextSpan(String textSpan)

given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.