Next: White Space and Comments Up: No Title Previous: Possible Problem

The Task of the Lexical Analyzer

The basic action of a lexical analyzer is to read some part of the input and to return a token at each call.
If there is no more input it returns a special EOF-token.
$\begin{program} Token~token;~~~~\=/$*$~last~token~read~$*$/ \\ Object~obj;~~~~~\... ...~~~~~~~~~~~~~~\\ gt set~pos~to~the~position~of~the~token~$*$/ \\ \}\end{program}$
In JLex nextToken will actually return an object of class Symbol containing token, obj, pos.
Throw away white space and comments in between tokens.
When does one token end and the next token start?

Christoph Zenger
3/23/2000