edu.illinois.cs.cogcomp.core.utilities

Class TokenUtils

java.lang.Object

edu.illinois.cs.cogcomp.core.utilities.TokenUtils

public class TokenUtils
extends Object

Utility functions related to tokens and tokenization.

Constructor Summary

Constructors

Constructor and Description
TokenUtils()

Method Summary

Modifier and Type	Method and Description
static IntPair[]	getTokenOffsets(String sentence, String[] tokens) Takes a string and its tokenizied form, and returns an array of span index pairs.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TokenUtils

public TokenUtils()

Method Detail

getTokenOffsets

public static IntPair[] getTokenOffsets(String sentence,
                                        String[] tokens)

Takes a string and its tokenizied form, and returns an array of span index pairs.

Parameters:

sentence - raw input text

tokens - the tokenized form of the sentence

Returns:

array of span index pairs.