Lexical Structure
Lexical Structure
This chapter describes the lexical structure of Rue programs, including tokens, comments, and whitespace.
The lexer processes source text and produces a sequence of tokens. Comments and whitespace are handled but do not produce tokens.
Maximal Munch
The lexer uses the maximal munch (or longest match) principle: at each position in the source text, the lexer consumes the longest sequence of characters that forms a valid token.
This principle resolves ambiguity when multiple token patterns could match at a position. For example, <= is lexed as a single <= token rather than < followed by =, and && is lexed as a single logical AND token rather than two & tokens.
fn main() -> i32 {
let x = 1 << 2; // << is a single left-shift token
let y = x <= 10; // <= is a single less-than-or-equal token
if true && false { 0 } else { 1 } // && is a single logical AND token
}
Source Encoding
Source text is encoded in UTF-8. The compiler reads each source file as a sequence of Unicode scalar values decoded from its UTF-8 bytes. A file whose bytes are not valid UTF-8 is rejected before lexing begins.
A Unicode scalar value outside the ASCII range (U+0080 and above) MAY appear only within a comment or a string literal. Everywhere else — in identifiers, keywords, numeric literals, operators, and delimiters — the lexer recognizes only ASCII characters, and a non-ASCII character encountered in such a position is a lexical error (E0001). Identifiers are therefore limited to ASCII letters, digits, and underscores (2.1), while the contents of a comment or string literal may be any UTF-8 text.
fn main() -> i32 {
// Non-ASCII text is fine in a comment: café résumé π
let s = "héllo, 世界"; // and inside a string literal
0
}