edu.illinois.cs.cogcomp.nlp.utilities

Class StringCleanup

java.lang.Object

edu.illinois.cs.cogcomp.nlp.utilities.StringCleanup

public class StringCleanup
extends Object

StringCleanup maps characters in "broader" encodings to comparable characters in "narrower" encodings -- e.g. from UTF-8 to ascii, or from non-UTF-8 to UTF-8 or smaller. This is useful for some NLP applications that don't handle wide characters or other "non-standard" inputs well (such as the Charniak parser).

Author:

mssammon

Constructor Summary

Constructors

Constructor and Description
StringCleanup()

Method Summary

Modifier and Type	Method and Description
static String	normalizeToAscii(String origString_)
static String	normalizeToEncoding(String origString_, Charset encoding_) tries to normalize string to specified encoding.
static String	normalizeToLatin1(String origString_)
static String	normalizeToUtf8(String origString_)
static String	removeControlCharacters(String origString_)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StringCleanup

public StringCleanup()

Method Detail

normalizeToEncoding

public static String normalizeToEncoding(String origString_,
                                         Charset encoding_)

tries to normalize string to specified encoding. The number of characters returned should be the same, and tokens should remain contiguous in the output; non-recognized characters will be substituted for *something*.

normalizeToUtf8

public static String normalizeToUtf8(String origString_)

normalizeToLatin1

public static String normalizeToLatin1(String origString_)

normalizeToAscii

public static String normalizeToAscii(String origString_)

removeControlCharacters

public static String removeControlCharacters(String origString_)