edu.illinois.cs.cogcomp.core.utilities

Class TextCleaner

java.lang.Object

edu.illinois.cs.cogcomp.core.utilities.TextCleaner

public class TextCleaner
extends Object

This class has methods for conversion of non-UTF-8 characters to UTF-8, and from UTF-8 to Ascii. These attempt to preserve character offsets, and to make intuitive replacements (see the StringCleanup class from coreUtilities).

Other methods try to remove problem character sequences from text to avoid known problems with NLP components for such sequences, but don't preserve character offsets.

Author:

mssammon

Constructor Summary

Constructors

Constructor and Description
TextCleaner(ResourceManager rm_)

Method Summary

Modifier and Type	Method and Description
static String	cleanDiscussionForumXml(StringTransformation xmlText, List<String> tagsWithText, List<String> tagsWithAtts, List<String> attributeNames) This class removes XML markup, for the most part.
String	cleanText(String text_) attempts to remove/replace characters likely to cause problems to NLP tools -- output should be ascii only
static void	main(String[] args) Test here.
static String	removeXmlLeaveAttributes(String origText, List<String> tagNames, List<String> attributeNames) This class removes XML markup, for the most part, but for the provided set of attribute names, it will leave their values in place and in the same position.
static String	replaceControlSequence(String origText_) noisy character sequences seen in some crawled text look like rendered control sequences; this method strips them all.
static String	replaceDuplicatePunctuation(String origText_) replaces duplicate punctuation with single punctuation
static String	replaceMisusedApostropheSymbol(String origText_) attempts to replace incorrect apostrophe symbol (after other substitutions for non-standard quotation marks)
static String	replaceTildesAndStars(String origText_) web documents sometimes use tildes and stars either for page section breaks or as bullets for bullet points; these may cause problems to NLP components
static String	replaceUnderscores(String origText_) replaces underscores with dashes (many crawled news articles seem to have substituted em- or en-dashes with underscores)
static StringTransformation	replaceXmlEscapedChars(StringTransformation xmlTextSt) given an xml string, replace xml-escaped characters with their ascii equivalent, padded with whitespace.
static String	replaceXmlTags(String origText)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

TextCleaner

public TextCleaner(ResourceManager rm_)
            throws SAXException,
                   IOException

Throws:

SAXException

IOException

Method Detail

replaceMisusedApostropheSymbol

public static String replaceMisusedApostropheSymbol(String origText_)

attempts to replace incorrect apostrophe symbol (after other substitutions for non-standard quotation marks)

replaceXmlTags

public static String replaceXmlTags(String origText)

removeXmlLeaveAttributes

public static String removeXmlLeaveAttributes(String origText,
                                              List<String> tagNames,
                                              List<String> attributeNames)

This class removes XML markup, for the most part, but for the provided set of attribute names, it will leave their values in place and in the same position. Content within quote tags is also replaced with white space (for the purpose of cleaning up the ERE code only).

Parameters:

origText - the original text markup.

tagNames - the names of tags containing the attributes leave, MUST BE LOWERCASE.

attributeNames - the names of attributes to leave, MUST BE LOWERCASE.

Returns:

the string.

cleanDiscussionForumXml

public static String cleanDiscussionForumXml(StringTransformation xmlText,
                                             List<String> tagsWithText,
                                             List<String> tagsWithAtts,
                                             List<String> attributeNames)

This class removes XML markup, for the most part. For specified tags that denote spans of text other than body text (e.g. quotes, headlines), the text value and offsets are reported. For specified tags and attributes, the attribute values and their offsets are reported. Content within quote tags is left in place (though quote tags are removed) and the offsets are reported with the other specified attributes.

Parameters:

xmlText - the original xml text.

tagsWithText - the names of tags containing text other than body text (e.g. headlines, quotes)

tagsWithAtts - the names of tags containing the attributes leave, MUST BE LOWERCASE.

attributeNames - the names of attributes to leave, MUST BE LOWERCASE.

Returns:

String comprising text.

replaceXmlEscapedChars

public static StringTransformation replaceXmlEscapedChars(StringTransformation xmlTextSt)

given an xml string, replace xml-escaped characters with their ascii equivalent, padded with whitespace.

Parameters:

xmlTextSt - StringTransformation containing text string to be modified

Returns:

StringTransformation passed in as arguments, with modifications

replaceUnderscores

public static String replaceUnderscores(String origText_)

replaces underscores with dashes (many crawled news articles seem to have substituted em- or en-dashes with underscores)

replaceTildesAndStars

public static String replaceTildesAndStars(String origText_)

web documents sometimes use tildes and stars either for page section breaks or as bullets for bullet points; these may cause problems to NLP components

this method strips these characters completely

replaceDuplicatePunctuation

public static String replaceDuplicatePunctuation(String origText_)

replaces duplicate punctuation with single punctuation

This sometimes happens due to a typo, or may be due to ad-hoc web formating -- in the latter case, this may not have the ideal effect.

In addition, use of double dashes and ellipses may cause problems to NLP components; this should help, though it may introduce extra sentence breaks in the case of ellipses.

replaceControlSequence

public static String replaceControlSequence(String origText_)

noisy character sequences seen in some crawled text look like rendered control sequences; this method strips them all.

main

public static void main(String[] args)
                 throws Exception

Test here.

Parameters:

args - not used.

Throws:

Exception

cleanText

public String cleanText(String text_)

attempts to remove/replace characters likely to cause problems to NLP tools -- output should be ascii only