Lexical Analysis / Syntactic Analysis

• The syntax of a programming language is given in two stages.
• Micro-Syntax describes the form of individual tokens (words).
• Macro-Syntax describes how programs are formed out of tokens
• The translation of source programs into token sequences is the main task of the lexical analyzer component in a compiler.
• Micro-Syntax is usually described by a regular language
• Hence, lexical analyzers can be finite state machines.
• For the Macro-Syntax finite state machines are not powerful enough. Programming languages are usually not regular.