edu.illinois.cs.cogcomp.nlp.corpusreaders

Class AbstractIncrementalCorpusReader<T>

java.lang.Object

edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader<T>

edu.illinois.cs.cogcomp.nlp.corpusreaders.AbstractIncrementalCorpusReader<T>

All Implemented Interfaces:

IResetableIterator<T>, Iterable<T>, Iterator<T>

Direct Known Subclasses:

XmlDocumentReader, XmlFragmentWhitespacingDocumentReader

public abstract class AbstractIncrementalCorpusReader<T>
extends AnnotationReader<T>

A corpus reader that allows incremental processing of a corpus (so not assuming all documents are read and instantiated on instantiating the reader). Goal is to return one TextAnnotation object at a time, where that TextAnnotation corresponds to a document or some other natural unit of text from the corpus. In addition, some non-body fields may be recorded as properties stored in the TextAnnotation object. These field names are arbitrary for now, but a CorpusReader must at least provide a list of names of these fields so that the user can retrieve them. The idea is that these will be somewhat corpus-specific, but will contain information useful for tasks related to the corpus -- e.g. article title/headline, byline, timestamp. TODO: establish some common fields/tags for this additional information. TODO: currently, assumes that each file in the corpus list will generate at least one TextAnnotation. Can avoid by pre-loading next file's annotations as soon as last TextAnnotation from current file is returned.

Field Summary

Fields

Modifier and Type	Field and Description
protected List<List<Path>>	fileList contains pointers to files comprising corpus.
protected String	sourceDirectory root directory of corpus

Fields inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader

corpusName, currentAnnotationId, resourceManager

Constructor Summary

Constructors

Constructor and Description
AbstractIncrementalCorpusReader(ResourceManager rm) ResourceManager must specify the fields CorpusReaderConfigurator.CORPUS_NAME and .CORPUS_DIRECTORY, plus whatever is required by the derived class for initializeReader().

Method Summary

Modifier and Type	Method and Description
String	generateReport() generate a human-readable report of annotations read from the source file (plus whatever other relevant statistics the user should know about).
abstract List<T>	getAnnotationsFromFile(List<Path> corpusFileListEntry) given an entry from the corpus file list generated by getFileListing() , parse its contents and get zero or more TextAnnotation objects.
abstract List<List<Path>>	getFileListing() generate a list of files comprising the corpus.
String	getSourceDirectory()
boolean	hasNext() is there another annotation object to return?
protected void	initializeReader() this method is called by the base class constructor, so all subclass-specific object initialization must be done here.
T	next() Returns the next element in the iteration.
void	reset() override this to conform to whatever the derived class's state mechanism requires.

Methods inherited from class edu.illinois.cs.cogcomp.nlp.corpusreaders.AnnotationReader

iterator, remove

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface java.lang.Iterable

forEach, spliterator

Methods inherited from interface java.util.Iterator

forEachRemaining

Field Detail

fileList

protected List<List<Path>> fileList

contains pointers to files comprising corpus. Each entry may consist of a source document (first element of each list) plus zero or more files with related annotations needed to instantiate the corresponding TextAnnotation objects.

sourceDirectory

protected String sourceDirectory

root directory of corpus

Constructor Detail

AbstractIncrementalCorpusReader

public AbstractIncrementalCorpusReader(ResourceManager rm)
                                throws Exception

ResourceManager must specify the fields CorpusReaderConfigurator.CORPUS_NAME and .CORPUS_DIRECTORY, plus whatever is required by the derived class for initializeReader().

Parameters:

rm - ResourceManager

Throws:

Exception

Method Detail

initializeReader

protected void initializeReader()

this method is called by the base class constructor, so all subclass-specific object initialization must be done here. This default implementation assumes that annotation and source are both provided in the same file.

Specified by:

initializeReader in class AnnotationReader<T>

reset

public void reset()

Description copied from class: AnnotationReader

override this to conform to whatever the derived class's state mechanism requires.

Specified by:

reset in interface IResetableIterator<T>

Overrides:

reset in class AnnotationReader<T>

getSourceDirectory

public String getSourceDirectory()

hasNext

public boolean hasNext()

Description copied from class: AnnotationReader

is there another annotation object to return?

Specified by:

hasNext in interface Iterator<T>

Specified by:

hasNext in class AnnotationReader<T>

Returns:

'true' if there is at least one more annotation, 'false' otherwise

next

public T next()

Returns the next element in the iteration.

Specified by:

next in interface Iterator<T>

Specified by:

next in class AnnotationReader<T>

Returns:

the next element in the iteration

Throws:

NoSuchElementException - if the iteration has no more elements

getFileListing

public abstract List<List<Path>> getFileListing()
                                         throws IOException

generate a list of files comprising the corpus. Each is expected to generate one or more TextAnnotation objects, though the way the iterator is implemented allows for corpus files to generate zero TextAnnotations if you are feeling picky. Note that the corpus directory is specified as a constructor argument.

Returns:

a list of lists of Path objects corresponding to files containing corpus documents to process. each list entry contains a set of files consisting of a source file that contains the base text that is decorated by annotations (plus possibly annotations as well) and zero or more files that contain annotations for that base text.

Throws:

IOException

getAnnotationsFromFile

public abstract List<T> getAnnotationsFromFile(List<Path> corpusFileListEntry)
                                        throws Exception

given an entry from the corpus file list generated by getFileListing() , parse its contents and get zero or more TextAnnotation objects.

Parameters:

corpusFileListEntry - corpus file containing content to be processed

Returns:

List of TextAnnotation objects extracted from the corpus file

Throws:

Exception

generateReport

public String generateReport()

generate a human-readable report of annotations read from the source file (plus whatever other relevant statistics the user should know about).

Specified by:

generateReport in class AnnotationReader<T>