edu.illinois.cs.cogcomp.tokenizer

Class EnglishTokenizer

java.lang.Object

edu.illinois.cs.cogcomp.tokenizer.EnglishTokenizer

All Implemented Interfaces:

Tokenizer

public class EnglishTokenizer
extends Object
implements Tokenizer

Created by ctsai12 on 12/13/15.

Nested Class Summary

Nested classes/interfaces inherited from interface edu.illinois.cs.cogcomp.nlp.tokenizer.Tokenizer

Tokenizer.Tokenization

Constructor Summary

Constructors

Constructor and Description
EnglishTokenizer()

Method Summary

Modifier and Type	Method and Description
TextAnnotation	getTextAnnotation(String text)
Pair<String[],IntPair[]>	tokenizeSentence(String sentence) given a sentence, return a set of tokens and their character offsets
Tokenizer.Tokenization	tokenizeTextSpan(String textSpan) given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

EnglishTokenizer

public EnglishTokenizer()

Method Detail

getTextAnnotation

public TextAnnotation getTextAnnotation(String text)

tokenizeSentence

public Pair<String[],IntPair[]> tokenizeSentence(String sentence)

given a sentence, return a set of tokens and their character offsets

Specified by:

tokenizeSentence in interface Tokenizer

Parameters:

sentence - The sentence string

Returns:

A Pair containing the array of tokens and their character offsets

tokenizeTextSpan

public Tokenizer.Tokenization tokenizeTextSpan(String textSpan)

given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Specified by:

tokenizeTextSpan in interface Tokenizer

Parameters:

textSpan -