edu.illinois.cs.cogcomp.ner

Class NerBenchmark

java.lang.Object

edu.illinois.cs.cogcomp.ner.NerBenchmark

public class NerBenchmark
extends Object

This runs a standard benchmark test across several datasets for the NER package. Training can be enabled by passing in the "-training" flag. The configurations and data are expected in the "benchmark" directory. Within, there will be a directory for each dataset you want to run against. Within those individual dataset directories, there will be a "config" directory containing one or more configuration files, a "test" directory (which can be a link to the directory with the test data), and a "train" directory where the training data files are found. If only a "train" directory is located in the directory, that will be used for both test and train directories. Each configuration file will result in a run to evaluate the results, and potentially one run to train up the model. The results are presented on standard out.

Directory format: - "benchmark" - <dataset name> there can be as many of these directories as you like, Reuters, Ontonotes, MUC7 and Web are examples of datasets one might run. - "config" : this must contain one or more configuration files, there will be a run per config file, only files ending with ".config" are processed - "test" : the test directory. If training, and not test directory, the "train" directory will be used for both. - "train" : the directory with the training data, only needed if "-training" passed. - "dev" : the hold out set for training. Command Line Options: -verbose : provide detailed report on all scoring methods with separate evaluation for phrase level tokenization, word level tokenization and so on. Alternatively, only the overall F1 scores are reported. -training : this option will cause a training run, if training, evaluation will not be performed, that requires another run. -features : for debugging, reports the feature vector for each token in the dataset. Output produced in a "features.out" file. -iterations : specify a fixed number of iterations, or -1 (the default) means auto converge requiring a "dev" directory. -release : build a final model for release, it will build on test and train, and unless "-iterations" specified, it will autoconvert using "dev" for a holdout set.

Constructor Summary

Constructors

Constructor and Description
NerBenchmark()

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args) Run a benchmark test against each subdirectory within the benchmark directory.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

NerBenchmark

public NerBenchmark()

Method Detail

main

public static void main(String[] args)
                 throws Exception

Run a benchmark test against each subdirectory within the benchmark directory.

Parameters:

args - may specify the directory containing the benchmarks.

Throws:

Exception - if anything goes wrong.