cpp.tokenize Namespace Reference

Classes
class	Token

Functions
def	GetTokens (source)
def	main (argv)

Variables
	VALID_IDENTIFIER_CHARS = set(_letters + _letters.upper() + '_0123456789$')
	HEX_DIGITS = set('0123456789abcdefABCDEF')
	INT_OR_FLOAT_DIGITS = set('01234567890eE-+')
string	UNKNOWN = 'UNKNOWN'
string	SYNTAX = 'SYNTAX'
string	CONSTANT = 'CONSTANT'
string	NAME = 'NAME'
string	PREPROCESSOR = 'PREPROCESSOR'
	WHENCE_STREAM
	WHENCE_QUEUE

Function Documentation

def cpp.tokenize.GetTokens

(

source

)

Returns a sequence of Tokens.

Args:
  source: string of C++ source code.

Yields:
  Token that represents the next token in the source.

References cpp.gmock_class.set.

Referenced by cpp.tokenize.main().

def cpp.tokenize.main

(

argv

)

Driver mostly for testing purposes.

References cpp.tokenize.GetTokens().

Variable Documentation

cpp.tokenize.VALID_IDENTIFIER_CHARS = set(_letters + _letters.upper() + '_0123456789$')

cpp.tokenize.HEX_DIGITS = set('0123456789abcdefABCDEF')

cpp.tokenize.INT_OR_FLOAT_DIGITS = set('01234567890eE-+')

string cpp.tokenize.UNKNOWN = 'UNKNOWN'

string cpp.tokenize.SYNTAX = 'SYNTAX'

string cpp.tokenize.CONSTANT = 'CONSTANT'

string cpp.tokenize.NAME = 'NAME'

Referenced by macro().

string cpp.tokenize.PREPROCESSOR = 'PREPROCESSOR'

cpp.tokenize.WHENCE_STREAM

cpp.tokenize.WHENCE_QUEUE