ANTLR3_COMMON_TOKEN_STREAM_struct (3) - Linux Manuals

NAME

ANTLR3_COMMON_TOKEN_STREAM_struct -

Common token stream is an implementation of ANTLR_TOKEN_STREAM for the default parsers and recognizers.

SYNOPSIS


#include <antlr3tokenstream.h>

Data Fields


ANTLR3_UINT32 channel

pANTLR3_LIST channelOverrides
Override map of tokens.
ANTLR3_BOOLEAN discardOffChannel
If this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table.
void(* discardOffChannelToks )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_BOOLEAN discard)
Signal to discard off channel tokens from here on in.
pANTLR3_LIST discardSet
Discared set.
void(* discardTokenType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 ttype)
Add a particular token type to the discard set.
void(* free )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM.
pANTLR3_LIST(* getTokenRange )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
Function that returns all the tokens between a start and a stop index.
pANTLR3_VECTOR(* getTokens )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)
Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet).
pANTLR3_LIST(* getTokensList )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)
Function that returns all the tokens indicated by being a member of the supplied List.
pANTLR3_LIST(* getTokensSet )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types)
Function that returns all the tokens indicated by the specified bitset, within a range of tokens.
pANTLR3_LIST(* getTokensType )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)
Function that returns all tokens of a certain type within a range.
ANTLR3_INT32 p
The index into the tokens list of the current token (the next one that will be consumed.
void(* setTokenTypeChannel )(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel)
A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel.
void * super
Whatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements.
pANTLR3_VECTOR tokens
Records every single token pulled from the source indexed by the token index.
pANTLR3_TOKEN_STREAM tstream
The ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation.

Detailed Description

Common token stream is an implementation of ANTLR_TOKEN_STREAM for the default parsers and recognizers.

You may of course build your own implementation if you are so inclined.

Field Documentation

ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_STREAM_struct::channel

Referenced by antlr3CommonTokenStreamSourceNew().

pANTLR3_LIST ANTLR3_COMMON_TOKEN_STREAM_struct::channelOverrides

Override map of tokens. If a token type has an entry in here, then the pointer in the table points to an int, being the override channel number that should always be used for this token type.

Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), and setTokenTypeChannel().

ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannel

If this flag is set to ANTLR3_TRUE, then tokens that the stream sees that are not in the channel that this stream is tuned to, are not tracked in the tokens table. When set to false, ALL tokens are added to the tracking.

Referenced by antlr3CommonTokenStreamSourceNew(), and discardOffChannel().

void(* ANTLR3_COMMON_TOKEN_STREAM_struct::discardOffChannelToks)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_BOOLEAN discard)

Signal to discard off channel tokens from here on in.

Referenced by antlr3CommonTokenStreamNew().

pANTLR3_LIST ANTLR3_COMMON_TOKEN_STREAM_struct::discardSet

Discared set. If a token has an entry in this table, then it is thrown away (data pointer is always NULL).

Referenced by antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), and discardTokenType().

void(* ANTLR3_COMMON_TOKEN_STREAM_struct::discardTokenType)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 ttype)

Add a particular token type to the discard set. If a token is found to belong to this set, then it is skipped/thrown away

Referenced by antlr3CommonTokenStreamNew().

void(* ANTLR3_COMMON_TOKEN_STREAM_struct::free)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)

Function that knows how to free an ANTLR3_COMMON_TOKEN_STREAM.

Referenced by antlr3CommonTokenStreamSourceNew().

pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokenRange)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)

Function that returns all the tokens between a start and a stop index. TODO: This is a new list (Ack! Maybe this is a reason to have factories for LISTS and HASHTABLES etc :-( come back to this)

Referenced by antlr3CommonTokenStreamNew().

pANTLR3_VECTOR(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokens)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream)

Function that returns a pointer to the ANTLR3_LIST of all tokens in the stream (this causes the buffer to fill if we have not get any yet).

Referenced by antlr3CommonTokenStreamNew().

pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensList)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_LIST list)

Function that returns all the tokens indicated by being a member of the supplied List.

Referenced by antlr3CommonTokenStreamNew().

pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensSet)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, pANTLR3_BITSET types)

Function that returns all the tokens indicated by the specified bitset, within a range of tokens.

Referenced by antlr3CommonTokenStreamNew(), getTokenRange(), getTokensList(), and getTokensType().

pANTLR3_LIST(* ANTLR3_COMMON_TOKEN_STREAM_struct::getTokensType)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop, ANTLR3_UINT32 type)

Function that returns all tokens of a certain type within a range.

Referenced by antlr3CommonTokenStreamNew().

ANTLR3_INT32 ANTLR3_COMMON_TOKEN_STREAM_struct::p

The index into the tokens list of the current token (the next one that will be consumed. p = -1 indicates that the token list is empty.

Referenced by antlr3CommonTokenStreamNew(), consume(), getTokens(), getTokensSet(), LB(), tokLT(), toString(), and toStringSS().

void(* ANTLR3_COMMON_TOKEN_STREAM_struct::setTokenTypeChannel)(struct ANTLR3_COMMON_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 ttype, ANTLR3_UINT32 channel)

A simple filter mechanism whereby you can tell this token stream to force all tokens of type ttype to be on channel. For example, when interpreting, we cannot exec actions so we need to tell the stream to force all WS and NEWLINE to be a different, ignored channel.

Referenced by antlr3CommonTokenStreamNew().

void* ANTLR3_COMMON_TOKEN_STREAM_struct::super

Whatever is supplying the COMMON_TOKEN_STREAM needs a pointer to itself so that this can be accessed by any of the API functions which it implements.

pANTLR3_VECTOR ANTLR3_COMMON_TOKEN_STREAM_struct::tokens

Records every single token pulled from the source indexed by the token index. There might be more efficient ways to do this, such as referencing directly in to the token factory pools, but for now this is convenient and the ANTLR3_LIST is not a huge overhead as it only stores pointers anyway, but allows for iterations and so on.

Referenced by antlr3CommonTokenStreamNew(), antlr3CTSFree(), consume(), get(), getTokens(), LB(), tokLT(), and toStringSS().

pANTLR3_TOKEN_STREAM ANTLR3_COMMON_TOKEN_STREAM_struct::tstream

The ANTLR3_TOKEN_STREAM interface implementation, which also includes the intstream implementation. We could duplicate the pANTLR_INT_STREAM in this interface and initialize it to a copy, but this could be confusing it just results in one more level of indirection and I think that with judicial use of 'const' later, the optimizer will do decent job.

Referenced by antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3CommonTokenStreamSourceNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), and getTokensSet().

Author

Generated automatically by Doxygen for ANTLR3C from the source code.

Pages related to ANTLR3_COMMON_TOKEN_STREAM_struct