`kong.lexer` — Tokenizer¶

kong.lexer.TOKEN_PATTERN = <_sre.SRE_Pattern object at 0x32e6fd0>¶: The re pattern that matches to tokens.

kong.lexer.tokenize(stream)¶

Makes tokens from input stream.

>>> t = lambda s: list(tokenize(s))
>>> t(u'a<-func  (123)')  
[kong.lexer.Token('identifier', u'a', 0),
 kong.lexer.Token('arrow', u'<-', 1),
 kong.lexer.Token('identifier', u'func', 3),
 kong.lexer.Token('space', u'  ', 7),
 kong.lexer.Token('parenthesis', u'(', 9),
 kong.lexer.Token('number', u'123', 10),
 kong.lexer.Token('parenthesis', u')', 13)]

It supports streaming as well:

>>> stream = [u'a(12', u'3)\nb<', u'-c * 123']
>>> t(stream)  
[kong.lexer.Token('identifier', u'a', 0),
 kong.lexer.Token('parenthesis', u'(', 1),
 kong.lexer.Token('number', u'123', 2),
 kong.lexer.Token('parenthesis', u')', 5),
 kong.lexer.Token('newline', u'\n', 6),
 kong.lexer.Token('identifier', u'b', 7),
 kong.lexer.Token('arrow', u'<-', 8),
 kong.lexer.Token('identifier', u'c', 10),
 kong.lexer.Token('space', u' ', 11),
 kong.lexer.Token('identifier', u'*', 12),
 kong.lexer.Token('space', u' ', 13),
 kong.lexer.Token('number', u'123', 14)]

Parameters:	stream (`collections.Iterable`) – input stream
Returns:	`Token` list
Return type:	`collections.Iterable`

class kong.lexer.Token(tag, string, offset)¶

A token that contains tag, string and offset.

tag¶: (basestring) The type of token e.g. 'arrow', 'colon'.

string¶: (basestring) The token string.

offset¶: (numbers.Integral) The token offset.

get_syntax_error(message=None)¶

Makes a SyntaxError with its offset.

Parameters:	message (`basestring`) – an optional error message
Returns:	an `SyntaxError` instance
Return type:	`SyntaxError`

exception kong.lexer.SyntaxError(offset, message=None)¶

An exception that rises when the syntax is invalid.

offset = None¶: (numbers.Integral) The errored offset of the string.

get_line(string)¶

Gets the errored line number from the code string.

Parameters:	string (`basestring`) – code string
Returns:	0-based line number
Return type:	`numbers.Integral`

get_column(string)¶

Gets the errored column number of the line from the code string.

Parameters:	string (`basestring`) – code string
Returns:	0-based column number
Return type:	`numbers.Integral`

kong.lexer — Tokenizer¶

`kong.lexer` — Tokenizer¶