edu.illinois.cs.cogcomp.nlp.corpusreaders

Class AbstractIncrementalXmlCorpusReader

java.lang.Object

edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalXmlCorpusReader

public abstract class AbstractIncrementalXmlCorpusReader
extends Object

A corpus reader that allows incremental processing of a corpus (so not assuming all documents are read and instantiated on instantiating the reader). Goal is to return one TextAnnotation object at a time, where that TextAnnotation corresponds to a document or some other natural unit of text from the corpus. In addition, some non-body fields may be recorded as properties stored in the TextAnnotation object. These field names are arbitrary for now, but a CorpusReader must at least provide a list of names of these fields so that the user can retrieve them. The idea is that these will be somewhat corpus-specific, but will contain information useful for tasks related to the corpus -- e.g. article title/headline, byline, timestamp. TODO: establish some common fields/tags for this additional information. TODO: currently, assumes that each file in the corpus list will generate at least one TextAnnotation. Can avoid by pre-loading next file's annotations as soon as last TextAnnotation from current file is returned.

Constructor Summary

Constructors

Constructor and Description
AbstractIncrementalXmlCorpusReader(ResourceManager rm) ResourceManager must specify the fields CorpusReaderConfigurator.CORPUS_NAME and .CORPUS_DIRECTORY, plus whatever is required by the derived class for initializeReader().

Method Summary

Modifier and Type	Method and Description
String	generateReport() generate a human-readable report of annotations read from the source file (plus whatever other relevant statistics the user should know about).
abstract List<List<Path>>	getFileListing() generate a list of files comprising the corpus.
String	getSourceDirectory()
abstract List<XmlTextAnnotation>	getXmlTextAnnotationsFromFile(List<Path> corpusFileListEntry) given an entry from the corpus file list generated by getFileListing() , parse its contents and get zero or more TextAnnotation objects.
boolean	hasNext()
protected void	initializeReader() this method is called by the base class constructor, so all subclass-specific object initialization must be done here.
XmlTextAnnotation	next() Returns the next element in the iteration.
void	reset()

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

AbstractIncrementalXmlCorpusReader

public AbstractIncrementalXmlCorpusReader(ResourceManager rm)
                                   throws Exception

ResourceManager must specify the fields CorpusReaderConfigurator.CORPUS_NAME and .CORPUS_DIRECTORY, plus whatever is required by the derived class for initializeReader().

Parameters:

rm - ResourceManager

Throws:

Exception

Method Detail

initializeReader

protected void initializeReader()

this method is called by the base class constructor, so all subclass-specific object initialization must be done here.

reset

public void reset()

getSourceDirectory

public String getSourceDirectory()

hasNext

public boolean hasNext()

next

public XmlTextAnnotation next()

Returns the next element in the iteration.

Returns:

the next element in the iteration

Throws:

NoSuchElementException - if the iteration has no more elements

getFileListing

public abstract List<List<Path>> getFileListing()
                                         throws IOException

generate a list of files comprising the corpus. Each is expected to generate one or more TextAnnotation objects, though the way the iterator is implemented allows for corpus files to generate zero TextAnnotations if you are feeling picky. Note that the corpus directory is specified as a constructor argument.

Returns:

a list of lists of Path objects corresponding to files containing corpus documents to process. each list entry contains a set of files consisting of a source file that contains the base text that is decorated by annotations (plus possibly annotations as well) and zero or more files that contain annotations for that base text.

Throws:

IOException

getXmlTextAnnotationsFromFile

public abstract List<XmlTextAnnotation> getXmlTextAnnotationsFromFile(List<Path> corpusFileListEntry)
                                                               throws Exception

given an entry from the corpus file list generated by getFileListing() , parse its contents and get zero or more TextAnnotation objects.

Parameters:

corpusFileListEntry - corpus file containing content to be processed

Returns:

List of TextAnnotation objects extracted from the corpus file

Throws:

Exception

generateReport

public String generateReport()

generate a human-readable report of annotations read from the source file (plus whatever other relevant statistics the user should know about).