antlr3-tokenSource (3) - Linux Manuals

Command to display antlr3-tokenSource manual in Linux: $ man 3 antlr3-tokenSource

NAME

ANTLR3_TOKEN_STREAM_struct -

Definition of the ANTLR3 common token stream interface.

SYNOPSIS

Data Fields

pANTLR3_COMMON_TOKEN(* _LT )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 k)
Get Token at current input pointer + i ahead where i=1 is next Token.
pANTLR3_DEBUG_EVENT_LISTENER debugger
Debugger interface, is this is a debugging token stream.
void(* free )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)
Function that knows how to free the memory for an ANTLR3_TOKEN_STREAM.
pANTLR3_COMMON_TOKEN(* get )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 i)
Get a token at an absolute index i; 0..n-1.
pANTLR3_TOKEN_SOURCE(* getTokenSource )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)
Where is this stream pulling tokens from? This is not the name, but a pointer into an interface that contains a ANTLR3_TOKEN_SOURCE interface.
ANTLR3_BOOLEAN initialStreamState
Indicates the initial stream state for dbgConsume().
pANTLR3_INT_STREAM istream
All input streams implement the ANTLR3_INT_STREAM interface.
void(* setDebugListener )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_DEBUG_EVENT_LISTENER debugger)
Function that sets the token stream into debugging mode.
void(* setTokenSource )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_TOKEN_SOURCE tokenSource)
Function that installs a token source for teh stream.
void * super
Whatever is providing this interface needs a pointer to itself so that this can be passed back to it whenever the api functions are called.
pANTLR3_TOKEN_SOURCE tokenSource
Pointer to the token source for this stream.
pANTLR3_STRING(* toString )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)
Return the text of all the tokens in the stream, as the old tramp in Leeds market used to say; 'Get the lot!'.
pANTLR3_STRING(* toStringSS )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)
Return the text of all tokens from start to stop, inclusive.
pANTLR3_STRING(* toStringTT )(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop)
Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location.

Detailed Description

Definition of the ANTLR3 common token stream interface.

Remarks:

: Much of the documentation for this interface is stolen from Ter's Java implementation.

Field Documentation

pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_STREAM_struct::_LT)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_INT32 k)

Get Token at current input pointer + i ahead where i=1 is next Token. i<0 indicates tokens in the past. So -1 is previous token and -2 is two tokens ago. LT(0) is undefined. For i>=n, return Token.EOFToken. Return null for LT(0) and any index that results in an absolute address that is negative.

Referenced by antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3RecognitionExceptionNew(), dbgConsume(), getMissingSymbol(), and setDebugListener().

pANTLR3_DEBUG_EVENT_LISTENER ANTLR3_TOKEN_STREAM_struct::debugger

Debugger interface, is this is a debugging token stream.

Referenced by antlr3CommonTokenDebugStreamSourceNew(), consumeInitialHiddenTokens(), dbgConsume(), and setDebugListener().

void(* ANTLR3_TOKEN_STREAM_struct::free)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)

Function that knows how to free the memory for an ANTLR3_TOKEN_STREAM.

Referenced by antlr3CTSFree(), and antlr3TokenStreamNew().

pANTLR3_COMMON_TOKEN(* ANTLR3_TOKEN_STREAM_struct::get)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 i)

Get a token at an absolute index i; 0..n-1. This is really only needed for profiling and debugging and token stream rewriting. If you don't want to buffer up tokens, then this method makes no sense for you. Naturally you can't use the rewrite stream feature. I believe DebugTokenStream can easily be altered to not use this method, removing the dependency.

Referenced by antlr3CommonTokenStreamNew(), consumeInitialHiddenTokens(), dbgConsume(), getTokensSet(), and toStringSS().

pANTLR3_TOKEN_SOURCE(* ANTLR3_TOKEN_STREAM_struct::getTokenSource)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)

Where is this stream pulling tokens from? This is not the name, but a pointer into an interface that contains a ANTLR3_TOKEN_SOURCE interface. The Token Source interface contains a pointer to the input stream and a pointer to a function that returns the next token.

Referenced by antlr3CommonTokenStreamNew(), and toStringSS().

ANTLR3_BOOLEAN ANTLR3_TOKEN_STREAM_struct::initialStreamState

Indicates the initial stream state for dbgConsume().

Referenced by antlr3CommonTokenDebugStreamSourceNew(), consumeInitialHiddenTokens(), dbgConsume(), dbgTokLT(), and setDebugListener().

pANTLR3_INT_STREAM ANTLR3_TOKEN_STREAM_struct::istream

All input streams implement the ANTLR3_INT_STREAM interface. ..

Referenced by alreadyParsedRule(), antlr3CommonTokenDebugStreamSourceNew(), antlr3CommonTokenStreamNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), consumeUntil(), consumeUntilSet(), dbgTokLT(), displayRecognitionError(), getTokensSet(), match(), matchAny(), memoize(), mismatch(), recover(), recoverFromMismatchedElement(), recoverFromMismatchedSet(), recoverFromMismatchedToken(), setDebugListener(), synpred(), tokLT(), toString(), and toStringSS().

void(* ANTLR3_TOKEN_STREAM_struct::setDebugListener)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_DEBUG_EVENT_LISTENER debugger)

Function that sets the token stream into debugging mode.

Referenced by antlr3CommonTokenStreamNew(), and setDebugListener().

void(* ANTLR3_TOKEN_STREAM_struct::setTokenSource)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_TOKEN_SOURCE tokenSource)

Function that installs a token source for teh stream.

Referenced by antlr3CommonTokenStreamNew(), and antlr3CommonTokenStreamSourceNew().

void* ANTLR3_TOKEN_STREAM_struct::super

Whatever is providing this interface needs a pointer to itself so that this can be passed back to it whenever the api functions are called.

Referenced by antlr3CommonTokenStreamNew(), antlr3CTSFree(), antlr3RecognitionExceptionNew(), consume(), get(), getMissingSymbol(), tokLT(), toString(), and toStringSS().

pANTLR3_TOKEN_SOURCE ANTLR3_TOKEN_STREAM_struct::tokenSource

Pointer to the token source for this stream.

Referenced by getTokenSource(), setTokenSource(), and tokLT().

pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toString)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream)

Return the text of all the tokens in the stream, as the old tramp in Leeds market used to say; 'Get the lot!'.

Referenced by antlr3CommonTokenStreamNew().

pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toStringSS)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, ANTLR3_UINT32 start, ANTLR3_UINT32 stop)

Return the text of all tokens from start to stop, inclusive. If the stream does not buffer all the tokens then it can just return an empty ANTLR3_STRING or NULL; Grammars should not access $ruleLabel.text in an action in that case.

Referenced by antlr3CommonTokenStreamNew(), toString(), and toStringTT().

pANTLR3_STRING(* ANTLR3_TOKEN_STREAM_struct::toStringTT)(struct ANTLR3_TOKEN_STREAM_struct *tokenStream, pANTLR3_COMMON_TOKEN start, pANTLR3_COMMON_TOKEN stop)

Because the user is not required to use a token with an index stored in it, we must provide a means for two token objects themselves to indicate the start/end location. Most often this will just delegate to the other toString(int,int). This is also parallel with the pTREENODE_STREAM->toString(Object,Object).