edu.illinois.cs.cogcomp.nlp.tokenizer

Class StatefulTokenizer

java.lang.Object

edu.illinois.cs.cogcomp.nlp.tokenizer.StatefulTokenizer

All Implemented Interfaces:

Tokenizer

public class StatefulTokenizer
extends Object
implements Tokenizer

This is the entry point to the tokenizer state machine. This class is thread-safe, the TokenizerStateMachine is not.

Author:

redman

Nested Class Summary

Nested classes/interfaces inherited from interface edu.illinois.cs.cogcomp.nlp.tokenizer.Tokenizer

Tokenizer.Tokenization

Constructor Summary

Constructors

Constructor and Description
StatefulTokenizer() Takes a boolean indicating if we are to split on dash or not.
StatefulTokenizer(boolean splitOnDash) Takes a boolean indicating if we are to split on dash or not.

Method Summary

Modifier and Type	Method and Description
boolean	isSplitOnDash()
void	setSplitOnDash(boolean splitOnDash)
Pair<String[],IntPair[]>	tokenizeSentence(String sentence) given a sentence, return a set of tokens and their character offsets
Tokenizer.Tokenization	tokenizeTextSpan(String textSpan) given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StatefulTokenizer

public StatefulTokenizer()

Takes a boolean indicating if we are to split on dash or not. The default constructor assumes we do split on dash.

StatefulTokenizer

public StatefulTokenizer(boolean splitOnDash)

Takes a boolean indicating if we are to split on dash or not. The default constructor assumes we do split on dash.

Parameters:

splitOnDash - if true, we will split words on a "-".

Method Detail

tokenizeSentence

public Pair<String[],IntPair[]> tokenizeSentence(String sentence)

Description copied from interface: Tokenizer

given a sentence, return a set of tokens and their character offsets

Specified by:

tokenizeSentence in interface Tokenizer

Parameters:

sentence - The sentence string

Returns:

A Pair containing the array of tokens and their character offsets

tokenizeTextSpan

public Tokenizer.Tokenization tokenizeTextSpan(String textSpan)

Description copied from interface: Tokenizer

given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Specified by:

tokenizeTextSpan in interface Tokenizer

isSplitOnDash

public boolean isSplitOnDash()

Returns:

the splitOnDash to split words when dash encounter or not.

setSplitOnDash

public void setSplitOnDash(boolean splitOnDash)

Parameters:

splitOnDash - the splitOnDash to set