edu.illinois.cs.cogcomp.core.utilities

Class StringTransformationCleanup

java.lang.Object

edu.illinois.cs.cogcomp.core.utilities.StringTransformationCleanup

public class StringTransformationCleanup
extends Object

StringCleanup maps characters in "broader" encodings to comparable characters in "narrower" encodings -- e.g. from UTF-8 to ascii, or from non-UTF-8 to UTF-8 or smaller. This is useful for some NLP applications that don't handle wide characters or other "non-standard" inputs well (such as the Charniak parser).

Author:

mssammon

Constructor Summary

Constructors

Constructor and Description
StringTransformationCleanup()

Method Summary

Modifier and Type	Method and Description
static Pair<Boolean,Character>	fixCharByType(int codepoint) Attempt to replace an out-of-charset character with something appropriate, so that the resulting string a) makes sense and b) resembles the original.
static StringTransformation	normalizeToAscii(StringTransformation origStringSt)
static StringTransformation	normalizeToEncoding(StringTransformation stringTransformation, Charset encoding) tries to normalize string to specified encoding.
static StringTransformation	normalizeToLatin1(StringTransformation origStringSt)
static StringTransformation	removeDiacritics(StringTransformation origStringSt)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StringTransformationCleanup

public StringTransformationCleanup()

Method Detail

normalizeToEncoding

public static StringTransformation normalizeToEncoding(StringTransformation stringTransformation,
                                                       Charset encoding)

tries to normalize string to specified encoding. The number of characters returned should be the same, and tokens should remain contiguous in the output; non-recognized characters will be substituted for *something*.

fixCharByType

public static Pair<Boolean,Character> fixCharByType(int codepoint)

Attempt to replace an out-of-charset character with something appropriate, so that the resulting string a) makes sense and b) resembles the original. Otherwise, use whitespace.

Parameters:

codepoint - codepoint of character to change

Returns:

flag indicating whether a substitution was found, and the substituted character.

removeDiacritics

public static StringTransformation removeDiacritics(StringTransformation origStringSt)

normalizeToLatin1

public static StringTransformation normalizeToLatin1(StringTransformation origStringSt)

normalizeToAscii

public static StringTransformation normalizeToAscii(StringTransformation origStringSt)