TokenizerStateMachine (illinois-cogcomp-nlp 3.1.29 API)

java.lang.Object
- edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine

```
public class TokenizerStateMachine
extends Object
```
This is the state machine used to parse text.

The parsing states are defined by the TokenizerState enumeration, the state transition set is defined by the TokenType enumeration. These two enumerations define the dimensions of the state machine matrix.

States are assumed to be nested rather than linear, so there is a state stack that contains State objects that encapsulate the state object as well as logic to manage poping and pushing that state. So the state objects do the work of actually creating the TextAnnotation objects.

The StateProcessor interface defines the interface class instances that operate on state transitions must adhere to to process new tokens as they are presented to the state machine
Potential issues:
1. URLs, when we encounter a ":", we check for protocol, if we find a protocol name following a space, we will skip to the next non white space, make the rest a url.
2. contractions with abbr, like can't, won't and those. if no spaces surround the "'", we will leave the word together including the "'" unless it is one of the common contractions like 's, 'm, 're, et cetera.
Author:

redman

Field Summary

Fields
Modifier and Type	Field and Description
`protected ArrayList<edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State>`	`completed` the state stack, since state can be nested.
`protected int`	`current` the character offset.
`protected ArrayList<edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State>`	`stack` the state stack, since state can be nested.
`protected int`	`state` the state we are in currently.
`protected StateProcessor[][]`	`statemachine` This is the state machine.
`protected char[]`	`text` the text to process.
`protected String`	`textstring` the text to process.

Constructor Summary

Constructors
Constructor and Description

TokenizerStateMachine(boolean splitOnDash)
Init the state machine decision matrix and the text annotation.

Constructors
Constructor and Description
`TokenizerStateMachine(boolean splitOnDash)` Init the state machine decision matrix and the text annotation.

Method Summary

All Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`protected edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State`	`getCurrent()` get the current state.
`protected boolean`	`isContinue()` Any number of periods beyond two will continue the sentence rather than ending it..
`protected boolean`	`isURL()` We have encountered a colon in the input data stream, check to see if it is a URL, and if it is, advance the cursor and return true, or return false.
`protected void`	`parseText(String intext)` Process the input text delineating sentences and words.
`protected char`	`peek(int offset)` get the character at the given offset from the current position.
`protected boolean`	`pop(int where)` Pop the current state identifier off the stack.
`protected void`	`push(edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State newState, int where)` Push a new state identifier off the stack.

Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

- Field Detail
  - stack
```
protected ArrayList<edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State> stack
```
    the state stack, since state can be nested.
  - completed
```
protected ArrayList<edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State> completed
```
    the state stack, since state can be nested.
  - statemachine
```
protected StateProcessor[][] statemachine
```
    This is the state machine. Cardinality of 1st dim indexed by tokenizer states(TokenizerState), 2nd dimension is indexed by of token types (TokenType enum). The values of this array are implementations of StateProcessor interface, are responsible for individually processing a character at a time, but they call also look ahead and back, as they have full access to the contents of this class.
    
    This class is initialized using Java's new lambda expressions, solely for the brevity, and considering the number of entries here, the expressiveness of a verbose interface implementation does not seem necessary. The StateProcessor interface has only one method, that method takes a char.
  - text
```
protected char[] text
```
    the text to process.
  - textstring
```
protected String textstring
```
    the text to process.
  - current
```
protected int current
```
    the character offset.
  - state
```
protected int state
```
    the state we are in currently.
- Constructor Detail
  - TokenizerStateMachine
```
public TokenizerStateMachine(boolean splitOnDash)
```
    Init the state machine decision matrix and the text annotation.
- Method Detail
  - isContinue
```
protected boolean isContinue()
```
    Any number of periods beyond two will continue the sentence rather than ending it..
    
    Returns:
    
    return true if continue previous sentence.
  - isURL
```
protected boolean isURL()
```
    We have encountered a colon in the input data stream, check to see if it is a URL, and if it is, advance the cursor and return true, or return false.
    
    Returns:
    
    return true if it is a url.
  - peek
```
protected char peek(int offset)
```
    get the character at the given offset from the current position.
    
    Parameters:
    
    offset - the offset from the current position.
    
    Returns:
    
    the next character that will be processed.
  - getCurrent
```
protected edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State getCurrent()
```
    get the current state.
    
    Returns:
    
    the current word.
  - pop
```
protected boolean pop(int where)
```
    Pop the current state identifier off the stack.
    
    Parameters:
    
    where - the position to terminate the previous token and start the new one.
    
    Returns:
    
    true if popped the last item off the stack.
  - push
```
protected void push(edu.illinois.cs.cogcomp.nlp.tokenizer.TokenizerStateMachine.State newState,
                    int where)
```
    Push a new state identifier off the stack.
    
    Parameters:
    
    newState - the new state to push.
    
    where - the start position.
  - parseText
```
protected void parseText(String intext)
```
    Process the input text delineating sentences and words. Purpose is to produce a TextAnnotation object, that requires these:
    - corpusId - this string identifier IDs the corpus of data.
    - id - this string identifies IDs the data.
    - text - this is the text.
    - characterOffsets - this array of integer pairs (IntPair[]) delineate the words.
    - tokens - This array of Strings (String[]) contains text tokens.
    - sentenceEndPositions - this array of integers(int[]) indicates the end of the sentence objects.
    Parameters:
    
    intext - the text to parse.

Class TokenizerStateMachine

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

stack

completed

statemachine

text

textstring

current

state

Constructor Detail

TokenizerStateMachine

Method Detail

isContinue

isURL

peek

getCurrent

pop

push

parseText