10.3. typped.predefined_token_sets

This module contains convenience functions for defining commonly-used groups of tokens. The functions are all copied to the PrattParser class as methods simply because the namespace is convenient. They can be used standalone or as methods of a parser instance.

typped.predefined_token_sets.def_default_whitespace(parser, space_label='k_space', space_regex='[ \\t]+', newline_label='k_newline', newline_regex='[\\n\\f\\r\\v]+', matcher_options=None)

Define the standard whitespace tokens for space and newline, setting them as ignored tokens.

typped.predefined_token_sets.def_default_single_char_tokens(parser, chars=None, exclude=None, make_literals=False)

The characters in the string chars are defined as tokens with default labels. Spaces are ignored in the string. If chars is not set then all the labels will be defined except those in the string exclude. If make_literals is true then the tokens will also be defined as token literals (via def_literal).

typped.predefined_token_sets.def_default_float_token(parser, token_label='k_float', signed=True, require_decimal=False, on_ties=0)

Define a token for floats with default label ‘k_float’. If signed is true (the default) then a leading ‘+’ or ‘-‘ is optionally part of the float. Otherwise the sign is not included. This is sometimes needed when the signs are defined as a prefix operators instead.

typped.predefined_token_sets.def_default_int_token(parser, token_label='k_int', signed=True, on_ties=0)

Define a token for ints with default label ‘k_int’. If signed is true (the default) then a leading ‘+’ or ‘-‘ is optionally part of the float. Otherwise the sign is not included.

typped.predefined_token_sets.def_default_identifier_token(parser, token_label='k_identifier', signed=True, on_ties=0)

Define a identifier. It is like Python identifiers: a letter or underscore followed by any number of letters, underscores, and digits.

typped.predefined_token_sets.def_multi_tokens(parser, tuple_list, **kwargs)

A convenience function, to define multiple tokens at once. Each element of the passed-in list should be a tuple containing the arguments to the ordinary def_token method. Calls the equivalent Lexer function.

typped.predefined_token_sets.def_multi_ignored_tokens(parser, tuple_list, **kwargs)

A convenience function, to define multiple ignored tokens at once. Each element of the passed-in list should be a tuple containing the arguments to the ordinary def_token method with ignore=True. Calls the equivalent Lexer function.