antlr3-skipToken (3) - Linux Manuals

NAME

ANTLR3_TOKEN_SOURCE_struct -

Definition of a token source, which has a pointer to a function that returns the next token (using a token factory if it is going to be efficient) and a pointer to an ANTLR3_INPUT_STREAM.

SYNOPSIS

#include <antlr3tokenstream.h>

Data Fields

ANTLR3_COMMON_TOKEN eofToken
A special pre-allocated token, which signifies End Of Tokens.
pANTLR3_STRING fileName
When the token source is constructed, it is populated with the file name from whence the tokens were produced by the lexer.
pANTLR3_COMMON_TOKEN(* nextToken )(struct ANTLR3_TOKEN_SOURCE_struct *tokenSource)
Pointer to a function that returns the next token in the stream.
ANTLR3_COMMON_TOKEN skipToken
A special pre-allocated token, which is returned by mTokens() if the lexer rule said to just skip the generated token altogether.
pANTLR3_STRING_FACTORY strFactory
Whoever is providing tokens, needs to provide a string factory too.
void * super
Whatever is supplying the token source interface, needs a pointer to itself so that this pointer can be passed to it when the nextToken function is called.

Detailed Description

Definition of a token source, which has a pointer to a function that returns the next token (using a token factory if it is going to be efficient) and a pointer to an ANTLR3_INPUT_STREAM.

This is slightly different to the Java interface because we have no way to implement multiple interfaces without defining them in the interface structure or casting (void *), which is too convoluted.

Field Documentation

ANTLR3_COMMON_TOKEN ANTLR3_TOKEN_SOURCE_struct::eofToken

A special pre-allocated token, which signifies End Of Tokens. Because this must be set up with the current input index and so on, we embed the structure and return the address of it. It is marked as factoryMade, so that it is never attempted to be freed.

Referenced by antlr3LexerNew(), nextTokenStr(), setCharStream(), and tokLT().

pANTLR3_STRING ANTLR3_TOKEN_SOURCE_struct::fileName

When the token source is constructed, it is populated with the file name from whence the tokens were produced by the lexer. This pointer is a copy of the one supplied by the CharStream (and may be NULL) so should not be manipulated other than to copy or print it.

Referenced by setCharStream().

pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_SOURCE_struct::nextToken)(struct ANTLR3_TOKEN_SOURCE_struct *tokenSource)

Pointer to a function that returns the next token in the stream.

Referenced by antlr3LexerNew().

ANTLR3_COMMON_TOKEN ANTLR3_TOKEN_SOURCE_struct::skipToken

A special pre-allocated token, which is returned by mTokens() if the lexer rule said to just skip the generated token altogether. Having this single token stops us wasting memory by have the token factory actually create something that we are going to SKIP(); anyway.

Referenced by antlr3LexerNew(), and nextTokenStr().

pANTLR3_STRING_FACTORY ANTLR3_TOKEN_SOURCE_struct::strFactory

Whoever is providing tokens, needs to provide a string factory too.

Referenced by antlr3LexerNew(), setCharStream(), and toStringSS().

void* ANTLR3_TOKEN_SOURCE_struct::super

Whatever is supplying the token source interface, needs a pointer to itself so that this pointer can be passed to it when the nextToken function is called.

Referenced by antlr3LexerNew(), nextToken(), and nextTokenStr().

Author

Generated automatically by Doxygen for ANTLR3C from the source code.