A simple lexer. You can use this class if you want to parse some data based on regular expressions. It is not really smart, as it is meant to be used for simple tasks. NQL uses it for example to parse queries. More...

#include <minilexer.hpp>

Inheritance diagram for Nuria::MiniLexer:

Public Types
typedef QPair< int, QString >	TokenValue

typedef QList< TokenValue >	TokenValueList

Public Member Functions
	MiniLexer (QObject *parent=0)

Qt::CaseSensitivity	matchSensitivity () const

void	setMatchSensitivity (Qt::CaseSensitivity value)

int	length () const

TokenValueList	tokenValueList () const

TokenValue	tokenValue (int at) const

const QString &	value (int at) const

int	token (int at) const

void	addRule (const QString &name, const QRegExp &regExp, int token)

void	addRule (const QString &name, const QString &string, int token)

void	addRule (const QString &name, const QRegularExpression &regExp, int token)

void	addDefinition (const QString &name, const QVariantList &def)

bool	lex (const QString &data)

QString	lastError () const

int	errorPosition () const

bool	hasStartDefinition () const

bool	hasRule (const QString &name) const

bool	hasDefinition (const QString &name) const

Static Public Member Functions
static MiniLexer *	createInstanceFromDefinition (const QString &definition, QString &error)

Detailed Description

A simple lexer. You can use this class if you want to parse some data based on regular expressions. It is not really smart, as it is meant to be used for simple tasks. NQL uses it for example to parse queries.

Warning: If you know Flex then be warned that the meanings of 'rules' and 'definitions' are switched in MiniLexer.

Note: If compiled on Qt5, QRegularExpression is also supported.

Example

Note: This example assumes that you have using namespace Nuria; somewhere in your code.

Lets see a little example. Lets say we want to write a INI-Parser. First, we need a MiniLexer instance. We also create a enumeration of possible tokens.

enum IniTokens { TokenGroup, TokenKey, TokenValue };

MiniLexer lexer;

Next we need some rules. A line in an ini file is either a group declaration or a Key=Value pair.
A group looks like this: [Groupname]
A Key=Value pair looks like: Key = Value
The whitespace in front and back of the equal sign is optional. So, our first rule will take care of groups:

// Rule for INI groups. Accepts anything in the brackets excluding [].

lexer.addRule ("Group", QRegExp ("^\\[[^\\[\\]]+\\]$"), TokenGroup);

We head directly to the next two rules:

// Matches on anything before the first equal sign
lexer.addRule ("Key", QRegExp ("[^=]+"), TokenKey);
// Matches on everything to the newline character
lexer.addRule ("Value", QRegExp ("[^\n]*", TokenValue);

Okay, now we need to tell MiniLexer how a line must look like. For this, we define two definitions. One for groups, and the other one for Key=Value pairs:

lexer.addDefinition (QString(), QVariantList() << QVariant::fromValue (LexerRule ("Group")));
lexer.addDefinition (QString(), QVariantList() << QVariant::fromValue (LexerRule ("Key"))
    << "=" << QVariant::fromValue (LexerRule ("Value")));

Next part, the lexing itself. We open up a file called "test.ini" using QFile and lex line after line, printing whatever we get. We break up if something goes wrong.

QFile file ("test.ini");
if (!file.open (QIODevice::ReadOnly | QIODevice::Text)) {
    qFatal ("Failed to open test.ini");
}
while (!file.atEnd ()) {
    if (!lexer.lex (file.readLine ())) {
        qFatal ("Syntax error.");
    }
    
    if (lexer.token (0) == TokenGroup) {
        qDebug() << "Group:" << lexer.value (0);
    } else {
        qDebug() << lexer.value (0) << "->" << lexer.value (1);
    }
}

And thats it. You could also use QSettings when you want to read ini files, but wheres the fun in it if you can also use a lexer? :P

Member Function Documentation

void Nuria::MiniLexer::addDefinition	(	const QString &	name,
		const QVariantList &	def
	)

Adds a definition to the lexer. A definition declares how the input should look like (Think of it being like BNF, but using C++ stuff to describe things instead of some text input). You can do that by using strings, regular expressions, rules and other definitions in the def list. You can add multiple definitions to a name. You can't remove a definition later. MiniLexer isn't really smart when it comes to matching. That being said, make sure that when a specific match order of definitions with the same name is important, MiniLexer tries from the top which is least-important to most-important at the bottom.

Note: MiniLexer uses "" as start definition. To add a start definition simply pass QString() as name.; To put a reference to a rule into the list, use: QVariant::fromValue(LexerRule("Name of the rule")); Same goes for references to definitions: QVariant::fromValue(LexerDefinition("Name of the definition"))

void Nuria::MiniLexer::addRule	(	const QString &	name,
		const QRegExp &	regExp,
		int	token
	)

Adds a rule to the lexer. name must be unique (if there is already a rule with the same name, it will be overwritten). regExp is the regular expressions which is connected to this rule. token is a user-defined token id which is used to distinguish between different rules after the lexer has been run.

Note: When you pass -1 as token, matches for this rule won't be stored for later use.; If you only care about a certain part of the match, then capture it using (brackets). MiniLexer will take the first captured text as value if it finds any. Else, it takes the whole match as value.

void Nuria::MiniLexer::addRule	(	const QString &	name,
		const QString &	string,
		int	token
	)

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

void Nuria::MiniLexer::addRule	(	const QString &	name,
		const QRegularExpression &	regExp,
		int	token
	)

This is an overloaded member function, provided for convenience. It differs from the above function only in what argument(s) it accepts.

static MiniLexer* Nuria::MiniLexer::createInstanceFromDefinition	(	const QString &	definition,
		QString &	error
	)

static

Creates a MiniLexer instance based on a definition string. If parsing failed, the hasStartDefinition() method of the returned MiniLexer instance will return false. error will contain a human-readable error message.
definition must be a string with the following format:

# Starts a comment. It ends at the end of the line. It can only appear at the beginning of a line
Rule(Token ): Body
- Rule = The name of the rule. May be prefixed with a $
- Token = The token id as integer
- Body = The rule body. May be a "string" or a /regex/
Definiton = Body ;
- Definition = The name of the definition. Must not be prefixed with a $
- Body = The body of the definition. The body contains one or more items separated with whitespace. The body ends with a semicolon ';'. Possible items are:
  - A C-style string literal enclosed in ""
  - A regular expression enclosed in // (Slashes). A single 'i' after the closing slash marks the regular expression to match case-insensitive.
  - A name of a definition. Just write the name itself.
  - A name of a rule. Prefix the name with a dollar-sign '$'

Note: You can format the definition string using different types of whitespace. The method is pretty free in that regard.; Also read the docs of addRule and addDefinition. They also apply on this function.

int Nuria::MiniLexer::errorPosition ( ) const

If lex fails, this function will return the position where the error occured.

bool Nuria::MiniLexer::hasDefinition ( const QString & name ) const

Returns true if there is a definition with the name name.

bool Nuria::MiniLexer::hasRule ( const QString & name ) const

Returns true if there is a rule with the name name.

bool Nuria::MiniLexer::hasStartDefinition ( ) const

Returns true if this instance has a start definition (e.g. a definition with the name "").

QString Nuria::MiniLexer::lastError ( ) const

If lex fails, this function will return a human-readable string describing the problem.

int Nuria::MiniLexer::length ( ) const

Returns the count of parsed tokens.

bool Nuria::MiniLexer::lex ( const QString & data )

Lexes data. Returns true if data could be completely parsed, returns false otherwise.

See also: tokenValueList tokenValue token value lastError

Qt::CaseSensitivity Nuria::MiniLexer::matchSensitivity ( ) const

Returns if strings are matched case-sensitive or case-insensitive.

See also: setMatchSensitivity

void Nuria::MiniLexer::setMatchSensitivity ( Qt::CaseSensitivity value )

Sets if strings should be matched case-sensitive or case-insensitive. This only applies to strings and has no effect on regular expressions. Default is case-sensitive matching.

int Nuria::MiniLexer::token ( int at ) const

Returns the token of a token/value pair at position at.

TokenValue Nuria::MiniLexer::tokenValue ( int at ) const

Returns a single TokenValue pair. TokenValue is a typedef for QPair<int,QString>. first is the token id, second the value.

TokenValueList Nuria::MiniLexer::tokenValueList ( ) const

Returns a list of TokenValue's containing the current list of token and values.

const QString& Nuria::MiniLexer::value ( int at ) const

Returns the value of a token/value pair at position at.

The documentation for this class was generated from the following file:

minilexer.hpp

Public Types

Public Member Functions

Static Public Member Functions

Detailed Description

Member Function Documentation