edu.illinois.cs.cogcomp.nlp.tokenizer

Class IllinoisTokenizer

java.lang.Object

edu.illinois.cs.cogcomp.nlp.tokenizer.IllinoisTokenizer

All Implemented Interfaces:

Tokenizer

Deprecated.

use StatefulTokenizer instead.

public class IllinoisTokenizer
extends Object
implements Tokenizer

Created by mssammon on 7/27/15.

Nested Class Summary

Nested classes/interfaces inherited from interface edu.illinois.cs.cogcomp.nlp.tokenizer.Tokenizer

Tokenizer.Tokenization

Constructor Summary

Constructors

Constructor and Description
IllinoisTokenizer() Deprecated.

Method Summary

Modifier and Type	Method and Description
Pair<String[],IntPair[]>	tokenizeSentence(String sentence) Deprecated. given a sentence, return a set of tokens and their character offsets
Tokenizer.Tokenization	tokenizeTextSpan(String text) Deprecated. given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

IllinoisTokenizer

public IllinoisTokenizer()

Deprecated.

Method Detail

tokenizeSentence

public Pair<String[],IntPair[]> tokenizeSentence(String sentence)

Deprecated.

given a sentence, return a set of tokens and their character offsets

Specified by:

tokenizeSentence in interface Tokenizer

Parameters:

sentence - the plain text sentence to tokenize

Returns:

an ordered list of tokens from the sentence, and an ordered list of their start and end character offsets (one-past-the-end indexing)

tokenizeTextSpan

public Tokenizer.Tokenization tokenizeTextSpan(String text)

Deprecated.

given a span of text, return a list of Pair< String[], IntPair[] > corresponding to tokenized sentences, where the String[] is the ordered list of sentence tokens and the IntPair[] is the corresponding list of character offsets with respect to the original text.

Specified by:

tokenizeTextSpan in interface Tokenizer

Parameters:

text - an arbitrary span of text.

Returns:

a Tokenization object containing the ordered token strings, their character offsets, and sentence end positions (as one-past-the-end token offsets)