edu.illinois.cs.cogcomp.tokenizer

Class MultilingualTokenizeTextToColumn

java.lang.Object

edu.illinois.cs.cogcomp.tokenizer.MultilingualTokenizeTextToColumn

public class MultilingualTokenizeTextToColumn
extends Object

Simple package to tokenize a file, or directory of files, and generate a new file/directory of files in column format.

Author:

mssammon

Constructor Summary

Constructors

Constructor and Description
MultilingualTokenizeTextToColumn(String lang)

Method Summary

Modifier and Type	Method and Description
static void	main(String[] args)
void	processDir(String corpusName, String inDir, String outDir) Process a directory containing plain text files (possibly in subdirectories).
void	processFile(String corpus, File in, String out) given an input containing plain text, tokenize and write to named output file.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

MultilingualTokenizeTextToColumn

public MultilingualTokenizeTextToColumn(String lang)

Method Detail

main

public static void main(String[] args)

processFile

public void processFile(String corpus,
                        File in,
                        String out)
                 throws IOException

given an input containing plain text, tokenize and write to named output file.

Parameters:

corpus - name of corpus

in - file to tokenize

out - output file for tokenized text

Throws:

IOException

processDir

public void processDir(String corpusName,
                       String inDir,
                       String outDir)
                throws IOException

Process a directory containing plain text files (possibly in subdirectories). Create a comparable structure rooted at outDir containing processed files. Create the root directory if it doesn't yet exist.

Parameters:

corpusName - name of corpus

inDir - directory of files to process

outDir - output directory for processed files

Throws:

IOException - if the input or output directories are invalid