EREDocumentReader (illinois-cogcomp-nlp 3.1.29 API)

java.lang.Object
- edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader<T>
- - edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader<XmlTextAnnotation>
  - - edu.illinois.cs.cogcomp.nlp.corpusreaders.XmlDocumentReader
    - - edu.illinois.cs.cogcomp.nlp.corpusreaders.ereReader.EREDocumentReader

All Implemented Interfaces:

IResetableIterator<XmlTextAnnotation>, Iterable<XmlTextAnnotation>, Iterator<XmlTextAnnotation>

Direct Known Subclasses:

ERENerReader
```
public class EREDocumentReader
extends XmlDocumentReader
```
Strips all useless XML markup from an ERE xml document leaving the original text and where needed, appropriate attribute values. Base class for other ERE readers for Named Entities, Mentions/Relations, and Events. There are THREE ERE English releases. Regrettably, they do not follow consistent standards for organization or for annotation. LDC2015E29_DEFT_Rich_ERE English V2 has two sets of annotation files: one, used for the Event Argument Extraction task in TAC that year, includes a small amount of additional markup to make each xml document well-formed. This changes the annotation offsets. Taggable entities within quoted blocks are annotated. LDC2015E68_DEFT_Rich_ERE_English R2_V2 has as source files excerpts from multi-post discussion forum documents. Taggable entities within quoted blocks are annotated. LDC2016E31_DEFT_Rich_ERE_English ENR3 has -- I believe -- complete threads, where annotation files may be broken into several chunks. Taggable entities within quoted blocks are NOT marked. TODO: handle DATELINE xml spans, which have content that probably should not be parsed as part of the text, but a value we'd like to retain

Author:

redman, msammon

Nested Class Summary

Nested Classes
Modifier and Type	Class and Description
`static class`	`EREDocumentReader.EreCorpus` prefix indicates language; suffix indicates release ENRX, ESRX, ZHRX are ERE releases in English, Spanish, and Chinese from LDC/DEFT KBPXX is a Knowledge Base Population corpus from year XX

Field Summary

Fields
Modifier and Type	Field and Description
`static String`	`ARG_ONE`
`static String`	`ARG_TWO`
`static String`	`AUTHOR`
`static String`	`CORPUS_TYPE`
`static String`	`DATELINE`
`static String`	`DATETIME`
`Set<String>`	`deletableSpanTags`
`static String`	`DOC`
`static String`	`ENTITIES` tags in ERE markup files
`static String`	`ENTITY`
`static String`	`ENTITY_ID`
`static String`	`ENTITY_MENTION`
`static String`	`ENTITY_MENTION_ID`
`static String`	`EntityHeadEndCharOffset`
`static String`	`EntityHeadStartCharOffset`
`static String`	`EntityIdAttribute`
`static String`	`EntityKbIdAttribute`
`static String`	`EntityMentionIdAttribute`
`static String`	`EntityMentionTypeAttribute` aim for consistent naming
`static String`	`EntitySpecificityAttribute`
`static String`	`EVENT_ARGUMENT`
`static String`	`EVENT_MENTION`
`static String`	`EventIdAttribute`
`static String`	`EventMentionIdAttribute`
`static String`	`FILL`
`static String`	`FILLER`
`static String`	`FILLER_ID`
`static String`	`FILLERS`
`static String`	`HEADLINE`
`static String`	`HOPPER`
`static String`	`HOPPERS`
`static String`	`ID`
`static String`	`IMG`
`static String`	`KBID`
`static String`	`LENGTH`
`static String`	`MENTION_HEAD`
`static String`	`MENTION_TEXT`
`static String`	`NAM`
`static String`	`NAME_END`
`static String`	`NAME_START`
`static String`	`NOM`
`static String`	`NOUN_TYPE`
`static String`	`OFFSET`
`static String`	`ORIG_AUTHOR`
`static String`	`ORIGIN`
`static String`	`POST`
`static String`	`PRO`
`static String`	`QUOTE` tags in document files
`static String`	`REALIS`
`static String`	`RELATION`
`static String`	`RELATION_MENTION`
`static String`	`RelationIdAttribute`
`static String`	`RelationMentionIdAttribute`
`static String`	`RelationRealisAttribute`
`static String`	`RELATIONS`
`static String`	`RelationSourceRoleAttribute`
`static String`	`RelationSubtypeAttribute`
`static String`	`RelationTargetRoleAttribute`
`static String`	`RelationTypeAttribute`
`static String`	`ROLE`
`static String`	`SARCASM`
`static String`	`SNIP`
`static String`	`SOURCE`
`static String`	`SPECIFICITY`
`static String`	`SQUISH`
`static String`	`STUFF`
`static String`	`SUBTYPE`
`Set<String>`	`tagsToIgnore`
`Map<String,Set<String>>`	`tagsWithAtts` tag sets for xml processor for ERE documents
`static String`	`TRIGGER`
`static String`	`TYPE`
`static String`	`UNKNOWN_KBID`
`static String`	`UNSPECIFIED`
`static String`	`WAYS`

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.XmlDocumentReader
fileId

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader
fileList, sourceDirectory

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader
corpusName, currentAnnotationId, resourceManager

Constructor Summary

Constructors
Constructor and Description
`EREDocumentReader(EREDocumentReader.EreCorpus ereCorpus, String corpusRoot, boolean throwExceptionOnXmlParseFailure)` build an EREDocumentReader configured for the specified ERE release.
`EREDocumentReader(EREDocumentReader.EreCorpus ereCorpus, TextAnnotationBuilder taBuilder, String corpusRoot, boolean throwExceptionOnXmlParseFailure)` build an EREDocumentReader configured for the specified ERE release, using provided TextAnnotationBuilder (allows for non-English, non-UIUC tokenizer)
`EREDocumentReader(ResourceManager rm, XmlTextAnnotationMaker xmlTextAnnotationMaker)`

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`static ResourceManager`	`buildEreConfig(String ereCorpusVal, String corpusRoot)` This method sets a range of configuration parameters based on which ERE release user specifies
`static XmlTextAnnotationMaker`	`buildEreXmlTextAnnotationMaker(String ereCorpusVal, boolean throwExceptionOnXmlParseFailure)` builds an XmlTextAnnotationMaker that handles the source files from the specified ERE release
`static XmlTextAnnotationMaker`	`buildXmlTextAnnotationMaker(EREDocumentReader.EreCorpus ereCorpus, boolean throwExceptionOnXmlParseFail)` builds an `XmlTextAnnotationMaker` for reading ERE format English corpus.
`static XmlTextAnnotationMaker`	`buildXmlTextAnnotationMaker(TextAnnotationBuilder textAnnotationBuilder, EREDocumentReader.EreCorpus ereCorpus, boolean throwExceptionOnXmlParseFail)` builds an `XmlTextAnnotationMaker` expecting ERE annotation.
`List<XmlTextAnnotation>`	`getAnnotationsFromFile(List<Path> corpusFileListEntry)` given an entry from the corpus file list generated by `getFileListing()` , parse its contents and get zero or more TextAnnotation objects.
`List<List<Path>>`	`getFileListing()` ERE corpus directory has two directories: source/ and ere/.
`static String`	`getPostViewName()`

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.XmlDocumentReader
generateReport, getRequiredAnnotationFileExtension, getRequiredSourceFileExtension, initializeReader, reset

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader
getSourceDirectory, hasNext, next

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader
iterator, remove

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable
forEach, spliterator

Methods inherited from interface java.util.Iterator
forEachRemaining

- Field Detail
  - QUOTE
```
public static final String QUOTE
```
    tags in document files
    
    See Also:
    
    Constant Field Values
  - AUTHOR
```
public static final String AUTHOR
```
    See Also:
    
    Constant Field Values
  - ID
```
public static final String ID
```
    See Also:
    
    Constant Field Values
  - DATETIME
```
public static final String DATETIME
```
    See Also:
    
    Constant Field Values
  - POST
```
public static final String POST
```
    See Also:
    
    Constant Field Values
  - DOC
```
public static final String DOC
```
    See Also:
    
    Constant Field Values
  - ORIG_AUTHOR
```
public static final String ORIG_AUTHOR
```
    See Also:
    
    Constant Field Values
  - HEADLINE
```
public static final String HEADLINE
```
    See Also:
    
    Constant Field Values
  - IMG
```
public static final String IMG
```
    See Also:
    
    Constant Field Values
  - SNIP
```
public static final String SNIP
```
    See Also:
    
    Constant Field Values
  - SQUISH
```
public static final String SQUISH
```
    See Also:
    
    Constant Field Values
  - STUFF
```
public static final String STUFF
```
    See Also:
    
    Constant Field Values
  - SARCASM
```
public static final String SARCASM
```
    See Also:
    
    Constant Field Values
  - ENTITIES
```
public static final String ENTITIES
```
    tags in ERE markup files
    
    See Also:
    
    Constant Field Values
  - ENTITY
```
public static final String ENTITY
```
    See Also:
    
    Constant Field Values
  - FILLERS
```
public static final String FILLERS
```
    See Also:
    
    Constant Field Values
  - FILLER
```
public static final String FILLER
```
    See Also:
    
    Constant Field Values
  - OFFSET
```
public static final String OFFSET
```
    See Also:
    
    Constant Field Values
  - TYPE
```
public static final String TYPE
```
    See Also:
    
    Constant Field Values
  - ENTITY_MENTION
```
public static final String ENTITY_MENTION
```
    See Also:
    
    Constant Field Values
  - NOUN_TYPE
```
public static final String NOUN_TYPE
```
    See Also:
    
    Constant Field Values
  - PRO
```
public static final String PRO
```
    See Also:
    
    Constant Field Values
  - NOM
```
public static final String NOM
```
    See Also:
    
    Constant Field Values
  - NAM
```
public static final String NAM
```
    See Also:
    
    Constant Field Values
  - FILL
```
public static final String FILL
```
    See Also:
    
    Constant Field Values
  - LENGTH
```
public static final String LENGTH
```
    See Also:
    
    Constant Field Values
  - MENTION_TEXT
```
public static final String MENTION_TEXT
```
    See Also:
    
    Constant Field Values
  - MENTION_HEAD
```
public static final String MENTION_HEAD
```
    See Also:
    
    Constant Field Values
  - SPECIFICITY
```
public static final String SPECIFICITY
```
    See Also:
    
    Constant Field Values
  - REALIS
```
public static final String REALIS
```
    See Also:
    
    Constant Field Values
  - RELATIONS
```
public static final String RELATIONS
```
    See Also:
    
    Constant Field Values
  - RELATION
```
public static final String RELATION
```
    See Also:
    
    Constant Field Values
  - RELATION_MENTION
```
public static final String RELATION_MENTION
```
    See Also:
    
    Constant Field Values
  - HOPPERS
```
public static final String HOPPERS
```
    See Also:
    
    Constant Field Values
  - HOPPER
```
public static final String HOPPER
```
    See Also:
    
    Constant Field Values
  - EVENT_MENTION
```
public static final String EVENT_MENTION
```
    See Also:
    
    Constant Field Values
  - EVENT_ARGUMENT
```
public static final String EVENT_ARGUMENT
```
    See Also:
    
    Constant Field Values
  - WAYS
```
public static final String WAYS
```
    See Also:
    
    Constant Field Values
  - SUBTYPE
```
public static final String SUBTYPE
```
    See Also:
    
    Constant Field Values
  - ARG_ONE
```
public static final String ARG_ONE
```
    See Also:
    
    Constant Field Values
  - ARG_TWO
```
public static final String ARG_TWO
```
    See Also:
    
    Constant Field Values
  - ENTITY_MENTION_ID
```
public static final String ENTITY_MENTION_ID
```
    See Also:
    
    Constant Field Values
  - ENTITY_ID
```
public static final String ENTITY_ID
```
    See Also:
    
    Constant Field Values
  - ROLE
```
public static final String ROLE
```
    See Also:
    
    Constant Field Values
  - FILLER_ID
```
public static final String FILLER_ID
```
    See Also:
    
    Constant Field Values
  - DATELINE
```
public static final String DATELINE
```
    See Also:
    
    Constant Field Values
  - CORPUS_TYPE
```
public static final String CORPUS_TYPE
```
    See Also:
    
    Constant Field Values
  - SOURCE
```
public static final String SOURCE
```
    See Also:
    
    Constant Field Values
  - TRIGGER
```
public static final String TRIGGER
```
    See Also:
    
    Constant Field Values
  - ORIGIN
```
public static final String ORIGIN
```
    See Also:
    
    Constant Field Values
  - UNKNOWN_KBID
```
public static final String UNKNOWN_KBID
```
    See Also:
    
    Constant Field Values
  - KBID
```
public static final String KBID
```
    See Also:
    
    Constant Field Values
  - EntityMentionTypeAttribute
```
public static final String EntityMentionTypeAttribute
```
    aim for consistent naming
    
    See Also:
    
    Constant Field Values
  - EntityIdAttribute
```
public static final String EntityIdAttribute
```
    See Also:
    
    Constant Field Values
  - EntityMentionIdAttribute
```
public static final String EntityMentionIdAttribute
```
    See Also:
    
    Constant Field Values
  - EntityHeadStartCharOffset
```
public static final String EntityHeadStartCharOffset
```
    See Also:
    
    Constant Field Values
  - EntityHeadEndCharOffset
```
public static final String EntityHeadEndCharOffset
```
    See Also:
    
    Constant Field Values
  - EntitySpecificityAttribute
```
public static final String EntitySpecificityAttribute
```
    See Also:
    
    Constant Field Values
  - RelationIdAttribute
```
public static final String RelationIdAttribute
```
    See Also:
    
    Constant Field Values
  - RelationMentionIdAttribute
```
public static final String RelationMentionIdAttribute
```
    See Also:
    
    Constant Field Values
  - RelationSubtypeAttribute
```
public static final String RelationSubtypeAttribute
```
    See Also:
    
    Constant Field Values
  - RelationTypeAttribute
```
public static final String RelationTypeAttribute
```
    See Also:
    
    Constant Field Values
  - RelationRealisAttribute
```
public static final String RelationRealisAttribute
```
    See Also:
    
    Constant Field Values
  - RelationSourceRoleAttribute
```
public static final String RelationSourceRoleAttribute
```
    See Also:
    
    Constant Field Values
  - RelationTargetRoleAttribute
```
public static final String RelationTargetRoleAttribute
```
    See Also:
    
    Constant Field Values
  - EventIdAttribute
```
public static final String EventIdAttribute
```
    See Also:
    
    Constant Field Values
  - EventMentionIdAttribute
```
public static final String EventMentionIdAttribute
```
    See Also:
    
    Constant Field Values
  - EntityKbIdAttribute
```
public static final String EntityKbIdAttribute
```
    See Also:
    
    Constant Field Values
  - NAME_START
```
public static final String NAME_START
```
    See Also:
    
    Constant Field Values
  - NAME_END
```
public static final String NAME_END
```
    See Also:
    
    Constant Field Values
  - UNSPECIFIED
```
public static final String UNSPECIFIED
```
    See Also:
    
    Constant Field Values
  - tagsWithAtts
```
public final Map<String,Set<String>> tagsWithAtts
```
    tag sets for xml processor for ERE documents
  - deletableSpanTags
```
public final Set<String> deletableSpanTags
```
  - tagsToIgnore
```
public final Set<String> tagsToIgnore
```
- Constructor Detail
  - EREDocumentReader
```
public EREDocumentReader(EREDocumentReader.EreCorpus ereCorpus,
                         TextAnnotationBuilder taBuilder,
                         String corpusRoot,
                         boolean throwExceptionOnXmlParseFailure)
                  throws Exception
```
    build an EREDocumentReader configured for the specified ERE release, using provided TextAnnotationBuilder (allows for non-English, non-UIUC tokenizer)
    
    Parameters:
    
    ereCorpus - a value from enum EreCorpus (e.g. 'ENR1', 'ENR2', or 'ENR3')
    
    taBuilder - TextAnnotationBuilder for target/language of choice
    
    throwExceptionOnXmlParseFailure -
    
    Throws:
    
    Exception
  - EREDocumentReader
```
public EREDocumentReader(EREDocumentReader.EreCorpus ereCorpus,
                         String corpusRoot,
                         boolean throwExceptionOnXmlParseFailure)
                  throws Exception
```
    build an EREDocumentReader configured for the specified ERE release.
    
    Parameters:
    
    ereCorpus - a value from enum EreCorpus (e.g. 'ENR1', 'ENR2', or 'ENR3')
    
    throwExceptionOnXmlParseFailure -
    
    Throws:
    
    Exception
  - EREDocumentReader
```
public EREDocumentReader(ResourceManager rm,
                         XmlTextAnnotationMaker xmlTextAnnotationMaker)
                  throws Exception
```
    Throws:
    
    Exception
- Method Detail
  - buildEreXmlTextAnnotationMaker
```
public static XmlTextAnnotationMaker buildEreXmlTextAnnotationMaker(String ereCorpusVal,
                                                                    boolean throwExceptionOnXmlParseFailure)
                                                             throws Exception
```
    builds an XmlTextAnnotationMaker that handles the source files from the specified ERE release
    
    Parameters:
    
    ereCorpusVal - a value corresponding to enum EreCorpus (e.g. 'ENR1', 'ENR2', or 'ENR3')
    
    throwExceptionOnXmlParseFailure - if 'true', xml reader will throw an exception if it finds e.g. mismatched xml tag open/close
    
    Returns:
    
    an XmlTextAnnotationMaker configured for the specified ERE corpus
    
    Throws:
    
    Exception
  - buildEreConfig
```
public static ResourceManager buildEreConfig(String ereCorpusVal,
                                             String corpusRoot)
                                      throws Exception
```
    This method sets a range of configuration parameters based on which ERE release user specifies
    
    Parameters:
    
    ereCorpusVal - a value corresponding to enum EreCorpus ('ENR1', 'ENR2', or 'ENR3')
    
    corpusRoot - the root directory of the corpus on your file system
    
    Returns:
    
    a ResourceManager with the appropriate configuration,
    
    Throws:
    
    Exception
  - buildXmlTextAnnotationMaker
```
public static XmlTextAnnotationMaker buildXmlTextAnnotationMaker(EREDocumentReader.EreCorpus ereCorpus,
                                                                 boolean throwExceptionOnXmlParseFail)
```
    builds an XmlTextAnnotationMaker for reading ERE format English corpus.
    
    Parameters:
    
    ereCorpus - which ERE release is being processed -- affects which tag blocks are marked
    
    throwExceptionOnXmlParseFail - if 'true', throw an exception if xml parser fails
    
    Returns:
    
    an XmlTextAnnotationMaker configured for English ERE.
  - buildXmlTextAnnotationMaker
```
public static XmlTextAnnotationMaker buildXmlTextAnnotationMaker(TextAnnotationBuilder textAnnotationBuilder,
                                                                 EREDocumentReader.EreCorpus ereCorpus,
                                                                 boolean throwExceptionOnXmlParseFail)
```
    builds an XmlTextAnnotationMaker expecting ERE annotation. TextAnnotationBuilder must be configured for the target language.
    
    Parameters:
    
    textAnnotationBuilder - a TextAnnotationBuilder with tokenizer suited to target language.
    
    throwExceptionOnXmlParseFail - if 'true', the XmlTextAnnotationMaker will throw an exception if any errors are found in the source xml.
    
    Returns:
    
    an XmlTextAnnotationMaker configured to parse an ERE corpus.
  - getPostViewName
```
public static String getPostViewName()
```
  - getFileListing
```
public List<List<Path>> getFileListing()
                                throws IOException
```
    ERE corpus directory has two directories: source/ and ere/. The source/ directory contains original text in an xml format. The ere/ directory contains markup files corresponding in a many-to-one relationship with the source/ files: related annotation files have the same prefix as the corresponding source file (up to the .xml suffix). (NOTE: release 1 (LDC2015E29) has two subdirectories for both source and annotation: one version is slightly modified by adding xml markup to make the source documents well-formed, which changes the annotation offsets. This method generates a List of List of Paths: each component List has the source file as its first element, and markup files as its remaining elements. It expects super.getSourceDirectory() to return the root directory of the ERE corpus, under which should be data/source/ and data/ere/ directories containing source files and annotation files respectively.
    
    Overrides:
    
    getFileListing in class XmlDocumentReader
    
    Returns:
    
    a list of Path objects corresponding to files containing corpus documents to process.
    
    Throws:
    
    IOException
  - getAnnotationsFromFile
```
public List<XmlTextAnnotation> getAnnotationsFromFile(List<Path> corpusFileListEntry)
                                               throws Exception
```
    given an entry from the corpus file list generated by getFileListing() , parse its contents and get zero or more TextAnnotation objects. This allows for the case where corpus annotations are provided in standoff format in one or more files separate from the source document. In such cases, the first file in the list should contain the source document and the rest should be the corresponding markup files. In this default implementation, it is assumed that a single file contains both source and markup.
    
    Overrides:
    
    getAnnotationsFromFile in class XmlDocumentReader
    
    Parameters:
    
    corpusFileListEntry - a list of files, the first of which is a source file.
    
    Returns:
    
    List of TextAnnotation objects extracted from the corpus file.
    
    Throws:
    
    Exception

Class EREDocumentReader

Nested Class Summary

Field Summary

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.XmlDocumentReader

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader

Constructor Summary

Method Summary

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.XmlDocumentReader

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader

Methods inherited from class java.lang.Object

Methods inherited from interface java.lang.Iterable

Methods inherited from interface java.util.Iterator

Field Detail

QUOTE

AUTHOR

ID

DATETIME

POST

DOC

ORIG_AUTHOR

HEADLINE

IMG

SNIP

SQUISH

STUFF

SARCASM

ENTITIES

ENTITY

FILLERS

FILLER

OFFSET

TYPE

ENTITY_MENTION

NOUN_TYPE

PRO

NOM

NAM

FILL

LENGTH

MENTION_TEXT

MENTION_HEAD

SPECIFICITY

REALIS

RELATIONS

RELATION

RELATION_MENTION

HOPPERS

HOPPER

EVENT_MENTION

EVENT_ARGUMENT

WAYS

SUBTYPE

ARG_ONE

ARG_TWO

ENTITY_MENTION_ID

ENTITY_ID

ROLE

FILLER_ID

DATELINE

CORPUS_TYPE

SOURCE

TRIGGER

ORIGIN

UNKNOWN_KBID

KBID

EntityMentionTypeAttribute

EntityIdAttribute

EntityMentionIdAttribute

EntityHeadStartCharOffset

EntityHeadEndCharOffset

EntitySpecificityAttribute

RelationIdAttribute

RelationMentionIdAttribute

RelationSubtypeAttribute

RelationTypeAttribute

RelationRealisAttribute

RelationSourceRoleAttribute

RelationTargetRoleAttribute