edu.illinois.cs.cogcomp.core.datastructures

Class Lexicon

java.lang.Object

edu.illinois.cs.cogcomp.core.datastructures.Lexicon

public class Lexicon
extends Object

A lexicon manager that manages features. Stores a hash value for string features and maps to an integer id. Optionally stores the string values too. Method previewFeature( String ) gets the

Author:

Vivek Srikumar

Field Summary

Fields

Modifier and Type	Field and Description
gnu.trove.map.hash.TIntIntHashMap	featureCounts
static String	GLOBAL_BIAS

Constructor Summary

Constructors

Constructor and Description
Lexicon(boolean hasBias, boolean storeStrings) Create a new lexicon object
Lexicon(InputStream in) Load a lexicon from the inputstream.
Lexicon(InputStream in, boolean loadStrings)

Method Summary

Modifier and Type	Method and Description
boolean	contains(String f)
void	countFeature(int featureId) Increment the count for featureId.
protected int	getFeatureHash(String featureName) A hash function from feature names to integers.
int	getFeatureId(String feature) a more intuitive method for adding a feature.
Pair<int[],float[]>	getFeatureVector(Map<String,Float> featureMap) generate a feature id representation from a feature vector with associated weights
Lexicon	getPrunedLexicon(int threshold) prunes the lexicon by removing features with less than threshold many counts
int	lookupId(String featureName) Get the internal id for this feature
String	lookupName(int id) Get the feature corresponding to the name.
void	previewFeature(String f) Add a new feature to this lexicon
Pair<int[],float[]>	pruneFeaturesByCount(int[] idx, float[] fs, int threshold)
void	save(String file) Saves the feature to id mapping.
int	size() The number of features in this lexicon
void	writeIntegerToFeatureStringFormat(PrintStream out)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

GLOBAL_BIAS

public static final String GLOBAL_BIAS

See Also:

Constant Field Values

featureCounts

public final gnu.trove.map.hash.TIntIntHashMap featureCounts

Constructor Detail

Lexicon

public Lexicon(boolean hasBias,
               boolean storeStrings)

Create a new lexicon object

Parameters:

hasBias - Include a default entry in the lexicon for GLOBAL_BIAS?

storeStrings - Store strings in the lexicon? Useful for debugging at the expense of much more memory consumption

Lexicon

public Lexicon(InputStream in)
        throws IOException

Load a lexicon from the inputstream. This does not load the strings into the lexicon, even if they are present.

Throws:

IOException

Lexicon

public Lexicon(InputStream in,
               boolean loadStrings)
        throws IOException

Throws:

IOException

Method Detail

lookupId

public int lookupId(String featureName)

Get the internal id for this feature

lookupName

public String lookupName(int id)

Get the feature corresponding to the name. Note: This function will throw a NullPointerException if the lexicon is not explicitly asked to keep the feature strings in memory. The default is not to keep strings in memory.

countFeature

public void countFeature(int featureId)

Increment the count for featureId.

getFeatureId

public int getFeatureId(String feature)

a more intuitive method for adding a feature. If already added, return id that was assigned; if not, add it with a unique id and return that id.

Parameters:

feature - Feature value to put in lexicon

Returns:

integer id for feature

previewFeature

public void previewFeature(String f)

Add a new feature to this lexicon

contains

public boolean contains(String f)

size

public int size()

The number of features in this lexicon

getFeatureHash

protected int getFeatureHash(String featureName)

A hash function from feature names to integers. The lexicon will lose features whenever there is a hash collision because it does not keep track of any of the strings.

getFeatureVector

public Pair<int[],float[]> getFeatureVector(Map<String,Float> featureMap)

generate a feature id representation from a feature vector with associated weights

Parameters:

featureMap -

Returns:

pruneFeaturesByCount

public Pair<int[],float[]> pruneFeaturesByCount(int[] idx,
                                                float[] fs,
                                                int threshold)

writeIntegerToFeatureStringFormat

public void writeIntegerToFeatureStringFormat(PrintStream out)
                                       throws IOException

Throws:

IOException

save

public void save(String file)
          throws IOException

Saves the feature to id mapping. Note: This does not store the feature names.

Throws:

IOException

getPrunedLexicon

public Lexicon getPrunedLexicon(int threshold)

prunes the lexicon by removing features with less than threshold many counts