edu.illinois.cs.cogcomp.core.utilities

Class XmlDocumentProcessor

java.lang.Object

edu.illinois.cs.cogcomp.core.utilities.XmlDocumentProcessor

public class XmlDocumentProcessor
extends Object

Reads an xml document from a corpus, extracts some fields as the document's clean NLP-processable text, and others as metadata. Provides a mapping from clean text character offsets to character offsets in the source xml. Metadata includes relevant character offsets. Some escaped xml elements are embedded in the document body text. These are made legible by substituting the escaped characters in an initial step. TODO: add constructor field to allow additional text clean/transform ops TODO: handle lone open/close tags: ideally, delete them, but at least transform them to avoid angle bracket parse problems in NLP components; retain relevant info like URL target etc.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	XmlDocumentProcessor.SpanInfo a structure to store span information: label, offsets, attributes (including value offsets)

Field Summary

Fields

Modifier and Type	Field and Description
static String	SPAN_INFO tag to indicate that the associated information denotes a span in source xml with the associated label.

Constructor Summary

Constructors

Constructor and Description
XmlDocumentProcessor(Set<String> deletableSpanTags, Map<String,Set<String>> tagsWithAtts, Set<String> singletonTags, boolean throwExceptionOnUnrecognizedTag) specify tags that determine processor behavior.

Method Summary

Modifier and Type	Method and Description
static Map<IntPair,Set<String>>	compileAttributeValues(List<XmlDocumentProcessor.SpanInfo> xmlMarkup) builds a map of attribute value offsets to attribute value to support search for metadata matching entity mentions
static Map<IntPair,XmlDocumentProcessor.SpanInfo>	compileOffsetSpanMapping(List<XmlDocumentProcessor.SpanInfo> retainedTagInfo) generate a mapping from span offset to span info
Pair<StringTransformation,List<XmlDocumentProcessor.SpanInfo>>	processXml(String xmlText) This class removes XML markup, for the most part.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

SPAN_INFO

public static final String SPAN_INFO

tag to indicate that the associated information denotes a span in source xml with the associated label.

See Also:

Constant Field Values

Constructor Detail

XmlDocumentProcessor

public XmlDocumentProcessor(Set<String> deletableSpanTags,
                            Map<String,Set<String>> tagsWithAtts,
                            Set<String> singletonTags,
                            boolean throwExceptionOnUnrecognizedTag)

specify tags that determine processor behavior.

Parameters:

deletableSpanTags - the names of tags containing text to be excluded from clean text (e.g. quotes)

tagsWithAtts - the names of tags containing the attributes to retain, paired with sets of attribute names MUST BE LOWERCASE.

singletonTags - the names of one-off tags (i.e. no close tag) whose contents must be skipped entirely

Method Detail

compileAttributeValues

public static Map<IntPair,Set<String>> compileAttributeValues(List<XmlDocumentProcessor.SpanInfo> xmlMarkup)

builds a map of attribute value offsets to attribute value to support search for metadata matching entity mentions

Parameters:

xmlMarkup - xml span information collected from source document

Returns:

a map from attribute value character offsets in source text to attribute value

compileOffsetSpanMapping

public static Map<IntPair,XmlDocumentProcessor.SpanInfo> compileOffsetSpanMapping(List<XmlDocumentProcessor.SpanInfo> retainedTagInfo)

generate a mapping from span offset to span info

Parameters:

retainedTagInfo - span information produced by XmlDocumentProcessor

Returns:

map from IntPair to SpanInfo

processXml

public Pair<StringTransformation,List<XmlDocumentProcessor.SpanInfo>> processXml(String xmlText)

This class removes XML markup, for the most part. For specified tags that denote spans of text other than body text (e.g. quotes, headlines), the text value and offsets are reported. For specified tags and attributes, the attribute values and their offsets are reported. Content within quote tags is left in place (though quote tags are removed) and the offsets are reported with the other specified attributes. This class has some facility for handling nested tags. Opens without closes are checked against tags to ignore (provided at construction) and if found are ignored (deleted). Otherwise, an exception is thrown.

Parameters:

xmlText - StringTransformation whose basis is the original xml text.

Returns:

String comprising text.