edu.illinois.cs.cogcomp.core.utilities

Class TextCleanerStringTransformation

java.lang.Object

edu.illinois.cs.cogcomp.core.utilities.TextCleanerStringTransformation

public class TextCleanerStringTransformation
extends Object

This class has methods for conversion of non-UTF-8 characters to UTF-8, and from UTF-8 to Ascii. These attempt to make intuitive replacements, and preserve a mapping to the original character offsets (see StringTransformationCleanup).

This class, which replicates functionality from TextCleaner, uses StringTransformations instead of Strings, and so allows recovery of character offsets in the original string after cleanup has taken place, even if character offsets are not preserved.

Author:

mssammon

Field Summary

Fields

Modifier and Type	Field and Description
protected static Pattern	adhocFormatPattern
protected static Pattern	atSymbolPattern
protected static Pattern	badApostrophePattern
protected static Pattern	controlSequencePattern
protected static Map<String,String>	escXlmToChar
protected static org.slf4j.Logger	logger
protected static Pattern	repeatPunctuationPattern
protected static Pattern	tagAttributePatter2 find attributes in an xml tag instance.
protected static Pattern	tagAttributePattern
protected static Pattern	underscorePattern
protected static Pattern	whitespacePattern
protected static String	xmlAmp
protected static String	xmlApos
protected static Pattern	xmlEscapeCharPattern
protected static String	xmlGt
protected static String	xmlLt
protected static String	xmlQuot
protected static Pattern	xmlTagNamePattern used to extract the name of the tag, so we can match tag and attribute name.
protected static Pattern	xmlTagPattern

Constructor Summary

Constructors

Constructor and Description
TextCleanerStringTransformation(ResourceManager rm)

Method Summary

Modifier and Type	Method and Description
StringTransformation	cleanText(StringTransformation origTextSt) attempts to remove/replace characters likely to cause problems to NLP tools -- output should be ascii only
protected static String	getSubstituteStr(String substr)
protected static boolean	keep(List<String> tagNames, List<String> attributeNames, String tagname, String attrName) determine if we should keep the value for this tag/attr.
static StringTransformation	replaceCharacters(StringTransformation origTextSt, Pattern pattern, String replacement) apply a matcher that finds instances of one or more sequences and replaces them with a specific string (including the empty string)
static StringTransformation	replaceDuplicatePunctuation(StringTransformation origTextSt) replaces duplicate punctuation with single punctuation
static StringTransformation	replaceMisusedApostropheSymbol(StringTransformation origTextSt) attempts to replace incorrect apostrophe symbol (after other substitutions for non-standard quotation marks)
static StringTransformation	replaceXmlEscapedChars(StringTransformation xmlTextSt) given an xml string, replace xml-escaped characters with their ascii equivalent, padded with whitespace.
static String	replaceXmlTags(String origText) identifies xml tags and replaces them with an equal amount of space characters.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

xmlQuot

protected static final String xmlQuot

See Also:

Constant Field Values

xmlAmp

protected static final String xmlAmp

See Also:

Constant Field Values

xmlApos

protected static final String xmlApos

See Also:

Constant Field Values

xmlLt

protected static final String xmlLt

See Also:

Constant Field Values

xmlGt

protected static final String xmlGt

See Also:

Constant Field Values

xmlEscapeCharPattern

protected static final Pattern xmlEscapeCharPattern

underscorePattern

protected static final Pattern underscorePattern

controlSequencePattern

protected static final Pattern controlSequencePattern

atSymbolPattern

protected static final Pattern atSymbolPattern

badApostrophePattern

protected static final Pattern badApostrophePattern

logger

protected static org.slf4j.Logger logger

repeatPunctuationPattern

protected static Pattern repeatPunctuationPattern

xmlTagPattern

protected static Pattern xmlTagPattern

xmlTagNamePattern

protected static Pattern xmlTagNamePattern

used to extract the name of the tag, so we can match tag and attribute name.

whitespacePattern

protected static Pattern whitespacePattern

tagAttributePatter2

protected static Pattern tagAttributePatter2

find attributes in an xml tag instance. Match whitespace then word. Group one is the attribute name, group 2 is the quote mark used, three is the value.

tagAttributePattern

protected static Pattern tagAttributePattern

adhocFormatPattern

protected static Pattern adhocFormatPattern

escXlmToChar

protected static Map<String,String> escXlmToChar

Constructor Detail

TextCleanerStringTransformation

public TextCleanerStringTransformation(ResourceManager rm)
                                throws SAXException,
                                       IOException

Throws:

SAXException

IOException

Method Detail

replaceMisusedApostropheSymbol

public static StringTransformation replaceMisusedApostropheSymbol(StringTransformation origTextSt)

attempts to replace incorrect apostrophe symbol (after other substitutions for non-standard quotation marks)

replaceXmlTags

public static String replaceXmlTags(String origText)

identifies xml tags and replaces them with an equal amount of space characters.

Parameters:

origText - text to clean

Returns:

cleaned text, same length, same non-whitespace character offsets

keep

protected static boolean keep(List<String> tagNames,
                              List<String> attributeNames,
                              String tagname,
                              String attrName)

determine if we should keep the value for this tag/attr.

Parameters:

tagNames - the names of the tags to keep.

attributeNames - the corresponding attributes within the above tag we want to keep.

tagname - the name of the current tag.

attrName - the name of the current attrigute.

Returns:

true if we keep the tag value.

replaceXmlEscapedChars

public static StringTransformation replaceXmlEscapedChars(StringTransformation xmlTextSt)

given an xml string, replace xml-escaped characters with their ascii equivalent, padded with whitespace.

Parameters:

xmlTextSt - StringTransformation containing text string to be modified

Returns:

StringTransformation passed in as arguments, with modifications

getSubstituteStr

protected static String getSubstituteStr(String substr)

replaceCharacters

public static StringTransformation replaceCharacters(StringTransformation origTextSt,
                                                     Pattern pattern,
                                                     String replacement)

apply a matcher that finds instances of one or more sequences and replaces them with a specific string (including the empty string)

Parameters:

origTextSt -

replaceDuplicatePunctuation

public static StringTransformation replaceDuplicatePunctuation(StringTransformation origTextSt)

replaces duplicate punctuation with single punctuation

This sometimes happens due to a typo, or may be due to ad-hoc web formating -- in the latter case, this may not have the ideal effect.

In addition, use of double dashes and ellipses may cause problems to NLP components; this should help, though it may introduce extra sentence breaks in the case of ellipses.

cleanText

public StringTransformation cleanText(StringTransformation origTextSt)

attempts to remove/replace characters likely to cause problems to NLP tools -- output should be ascii only