Class Tokenizer

java.lang.Object
  Tokenizer

public class Tokenizer
extends java.lang.Object

Tokenizer takes a text file and parses it into individual tokens, returning each as a string. The expected format of the text file is to have comments at the top of the file, in (* ... *) format, which are ignored by the Tokenizer. The tokens then follow, one per line.

Version:

7/30/08

Author:

Chris Fernandes

Constructor Summary

Tokenizer(java.lang.String filename)
Constructor specifying the file to tokenize.

Method Summary

java.lang.String

nextToken() Returns the next token in the input file.

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer(java.lang.String filename)

Constructor specifying the file to tokenize. Anything between (* and *) at the top of the file will be ignored, and line terminators are delimiters.

Parameters:

filename - absolute or relative path to the file to be tokenized. Examples: "input.txt" or "C:/input.txt"

Method Detail

nextToken

public java.lang.String nextToken()

Returns the next token in the input file.

Returns:

the next unread token. Returns null when end-of-file is reached