public class TokenizerUtilities extends Object
Modifier and Type | Class and Description |
---|---|
static class |
TokenizerUtilities.SentenceViewGenerators
The elements of this enumeration generate views called
SENTENCE . |
Modifier and Type | Field and Description |
---|---|
static Tokenizer |
lbjTokenizer |
static Tokenizer |
whiteSpaceTokenizer |
Constructor and Description |
---|
TokenizerUtilities() |
Modifier and Type | Method and Description |
---|---|
static SpanLabelView |
addTokenView(TextAnnotation input,
Tokenizer tokenizer,
String source) |
static edu.illinois.cs.cogcomp.core.datastructures.IntPair[] |
getTokenOffsets(String sentence,
String[] tokens) |
public static final Tokenizer lbjTokenizer
public static final Tokenizer whiteSpaceTokenizer
public static edu.illinois.cs.cogcomp.core.datastructures.IntPair[] getTokenOffsets(String sentence, String[] tokens)
public static SpanLabelView addTokenView(TextAnnotation input, Tokenizer tokenizer, String source)
Copyright © 2015. All rights reserved.