Nuria::TokenizerRules Class Reference

Storage of rules used by Nuria::Tokenizer. More...

#include <tokenizer.hpp>

Public Types
typedef std::function< bool(Token &, Tokenizer *) >	TokenAction
enum	WhitespaceMode { AutoHandleWhitespace = 0, ManualWhitespaceHandling }

Public Member Functions
	TokenizerRules (WhitespaceMode mode=AutoHandleWhitespace)
	TokenizerRules (const TokenizerRules &other)
	~TokenizerRules ()
void	addRegexToken (int tokenId, const QByteArray &regularExpression)
void	addRegexToken (int tokenId, const std::regex &regularExpression)
void	addStringToken (int tokenId, const QByteArray &terminal)
TokenizerRules &	operator= (const TokenizerRules &other)
void	setTokenAction (int tokenId, TokenAction action)
void	setWhitespaceMode (WhitespaceMode mode)
WhitespaceMode	whitespaceMode () const

Friends
class	Tokenizer

Detailed Description

Storage of rules used by Nuria::Tokenizer.

A instance of this class define a rule-set for Nuria::Tokenizer, defining how to read the data stream.

Rules are not directly stored in Nuria::Tokenizer to allow for complex grammatics, demanding for multiple rule-sets. Please see Nuria::Tokenizer for usage details.

Note

This structure is implicitly shared.

Precedence

During matching, string tokens are tested first, meaning that string tokens take precedence over regular-expression ones. Matches are tried in the order they were added to the rule-set, meaning the first added rule will also first tried.

Whitespace handling

TokenizerRules can automatically handle whitespace, meaning that it's automatically skipped. This is the default setting.

Member Typedef Documentation

typedef std::function< bool(Token &, Tokenizer *) > Nuria::TokenizerRules::TokenAction

Typedef of a token action. See setTokenAction() for details.

Member Enumeration Documentation

enum Nuria::TokenizerRules::WhitespaceMode

Whitespace handling modes.

Enumerator
AutoHandleWhitespace	Automatically ignore whitespace. Default setting.
ManualWhitespaceHandling	Do not ignore whitespace.

Constructor & Destructor Documentation

Nuria::TokenizerRules::TokenizerRules

(

WhitespaceMode

mode = AutoHandleWhitespace

)

Constructs a empty rule-set.

Nuria::TokenizerRules::TokenizerRules

(

const TokenizerRules &

other

)

Copy constructor.

Nuria::TokenizerRules::~TokenizerRules

(

)

Destructor.

Member Function Documentation

void Nuria::TokenizerRules::addRegexToken	(	int	tokenId,
		const QByteArray &	regularExpression
	)

Adds a token matching regularExpression. This method is provided for convenience. It's equivalent to:

addRegexToken (tokenId, QRegularExpression (regularExpression));

void Nuria::TokenizerRules::addRegexToken	(	int	tokenId,
		const std::regex &	regularExpression
	)

Adds a token matching the regularExpression.

void Nuria::TokenizerRules::addStringToken	(	int	tokenId,
		const QByteArray &	terminal
	)

Adds a token matching exactly terminal.

TokenizerRules& Nuria::TokenizerRules::operator=

(

const TokenizerRules &

other

)

Assignment operator.

void Nuria::TokenizerRules::setTokenAction	(	int	tokenId,
		TokenAction	action
	)

Sets action as handler for all tokens of type tokenId. The handler will be called everytime a token of that type is encountered. The prototype of action is as follows:

bool handler (Token &token, Tokenizer *tokenizer);

The handler will be passed the token itself as mutable reference and the Tokenizer instance. On success, true must be returned by the handler. If the handler returns false, tokenizing will fail.

The handler may change the contents of the passed token. If the handler changes the tokenId of the token, the token action handler of that new tokenId will not be invoked.

void Nuria::TokenizerRules::setWhitespaceMode

(

WhitespaceMode

mode

)

Sets the whitespace handling mode.

WhitespaceMode Nuria::TokenizerRules::whitespaceMode

(

)

const

Returns the current whitespace handling mode.

---

The documentation for this class was generated from the following file:

• tokenizer.hpp