ANTLR3_COMMON_TOKEN_struct (3) - Linux Manuals
NAME
ANTLR3_COMMON_TOKEN_struct -
The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.
SYNOPSIS
#include <antlr3commontoken.h>
Data Fields
The definition of an ANTLR3 common token structure, which all implementations of a token stream should provide, installing any further structures in the custom pointer element of this structure.
Remarks:
The virtual channel that this token exists in.
Referenced by antlr3SetTokenAPI(), emit(), getChannel(), and setChannel().
The character position in the line that this token was derived from.
Referenced by emit(), getCharPositionInLine(), and setCharPositionInLine().
Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar. No sense in constructing a pANTLR3_STRING just for that, as mostly the text will not be accessed - if it is, then we will build a pANTLR3_STRING for it a that point.
Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), getText(), and setText8().
Pointer to a custom element that the ANTLR3 programmer may define and install.
Referenced by antlr3SetTokenAPI(), emit(), factoryClose(), and getMissingSymbol().
Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.
Referenced by antlr3LexerNew(), antlr3TokenFactoryNew(), newPoolToken(), newToken(), and nextTokenStr().
Pointer to a function that knows how to free the custom structure when the token is destroyed.
Referenced by antlr3SetTokenAPI(), and factoryClose().
Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), serializeToken(), and toString().
Pointer to a function that gets the offset in the line where this token exists.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getCharPositionInLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().
Pointer to a function that gets the 'line' number where this token resides.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getLine(), getMissingSymbol(), serializeNode(), serializeToken(), and toString().
Pointer to a function that gets the start index in the input stream for this token.
Referenced by antlr3SetTokenAPI(), getText(), and toString().
Pointer to a function that gets the stop index in the input stream for this token.
Referenced by antlr3SetTokenAPI(), getText(), and toString().
Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), serializeToken(), toString(), and toStringSS().
Pointer to a function that returns an index 0...n-1 of the token in the token input stream.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), dbgSetTokenBoundaries(), serializeToken(), setTokenBoundaries(), toString(), and toStringTT().
Pointer to a function that returns the token type of this token.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), getTokensSet(), getType(), and serializeToken().
What the index of this token is, 0, 1, . ., n-2, n-1 tokens
Referenced by antlr3SetTokenAPI(), getTokenIndex(), and setTokenIndex().
Pointer to the input stream that this token originated in.
Referenced by antlr3RecognitionExceptionNew(), antlr3SetTokenAPI(), createToken(), getMissingSymbol(), getStartIndex(), getText(), newPoolToken(), and setInputStream().
The line number in the input stream where this token was derived from.
Referenced by antlr3SetTokenAPI(), emit(), getLine(), and setLine().
The offset into the input stream that the line in which this token resides starts.
Referenced by emit(), and getMissingSymbol().
Pointer to a function that sets the channel that this token should belong to.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().
Pointer to a function that sets the offset in the line where this token exists.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and getMissingSymbol().
Pointer to a function that sets the 'line' number where this token reside.
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), getMissingSymbol(), and nextTokenStr().
Pointer to a function that sets the start index in the input stream for this token.
Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().
Pointer to a function that sets the stop index in the input stream for this token.
Referenced by antlr3SetTokenAPI(), nextTokenStr(), and tokLT().
Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actaully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.
Referenced by antlr3SetTokenAPI().
Pointer to a function that 'might' be able to set the text associated with a token. Imaginary tokens such as an ANTLR3_CLASSIC_TOKEN may actually do this, however many tokens such as ANTLR3_COMMON_TOKEN do not actully have strings associated with them but just point into the current input stream. These tokens will implement this function with a function that errors out (probably drastically.
Referenced by antlr3SetTokenAPI(), createTypeTokenText(), and getMissingSymbol().
Pointer to a function that can set the token index of this token in the token input stream.
Referenced by antlr3SetTokenAPI(), and createTokenFromToken().
Pointer to a function that sets the type of this token.
Referenced by antlr3CommonTokenNew(), antlr3LexerNew(), antlr3SetTokenAPI(), createToken(), createTokenFromToken(), createTypeToken(), createTypeTokenText(), and getMissingSymbol().
The character offset in the input stream where the text for this token starts.
Referenced by antlr3SetTokenAPI(), emit(), getStartIndex(), and setStartIndex().
The character offset in the input stream where the text for this token stops.
Referenced by antlr3SetTokenAPI(), emit(), getStopIndex(), and setStopIndex().
A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.
Referenced by ANTLR3_TREE_ADAPTORNew(), antlr3CommonTreeNodeStreamNew(), antlr3CommonTreeNodeStreamNewStream(), antlr3LexerNew(), createToken(), getText(), newPoolToken(), setCharStream(), and setInputStream().
Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.
Referenced by createTokenFromToken(), emit(), getText(), setText(), and setText8().
Indicates whether this token is carrying:. State | Meaning ------+-------------------------------------- 0 | Nothing (neither rewrite text, nor setText) 1 | char * to user supplied rewrite text 2 | pANTLR3_STRING because of setText or similar action
Affects the union structure tokText below (uses 32 bit so alignment is always good)
Referenced by antlr3CommonTreeNodeStreamNew(), antlr3SetTokenAPI(), createToken(), createTokenFromToken(), emit(), getText(), setText(), and setText8().
Referenced by antlr3CommonTreeNodeStreamNew(), createToken(), createTokenFromToken(), emit(), getText(), setText(), and setText8().
Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
Referenced by antlr3SetTokenAPI(), createTokenFromToken(), and displayRecognitionError().
The actual type of this token.
Referenced by antlr3SetTokenAPI(), displayRecognitionError(), emit(), getText(), getType(), nextToken(), setType(), and toString().
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.
Referenced by antlr3SetTokenAPI(), emit(), and getMissingSymbol().
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.
Referenced by antlr3SetTokenAPI(), emit(), and getMissingSymbol().
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token. This is one of them.
Referenced by antlr3SetTokenAPI(), emit(), and getMissingSymbol().
Generated automatically by Doxygen for ANTLR3C from the source code.
ANTLR3_UINT32 channel
The virtual channel that this token exists in.
ANTLR3_INT32 charPosition
The character position in the line that this token was derived from.
void * custom
Pointer to a custom element that the ANTLR3 programmer may define and install.
ANTLR3_BOOLEAN factoryMade
Indicates that a token was produced from the token factory and therefore the the freeToken() method should not do anything itself because token factory is responsible for deleting it.
void(* freeCustom )(void *custom)
Pointer to a function that knows how to free the custom structure when the token is destroyed.
ANTLR3_UINT32(* getChannel )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the channel that this token was placed in (parsers can 'tune' to these channels.
ANTLR3_INT32(* getCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the offset in the line where this token exists.
ANTLR3_UINT32(* getLine )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the 'line' number where this token resides.
ANTLR3_MARKER(* getStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the start index in the input stream for this token.
ANTLR3_MARKER(* getStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that gets the stop index in the input stream for this token.
pANTLR3_STRING(* getText )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to function that returns the text pointer of a token, use toString() if you want a pANTLR3_STRING version of the token.
ANTLR3_MARKER(* getTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns an index 0...n-1 of the token in the token input stream.
ANTLR3_UINT32(* getType )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns the token type of this token.
ANTLR3_MARKER index
What the index of this token is, 0, 1, .
pANTLR3_INPUT_STREAM input
Pointer to the input stream that this token originated in.
ANTLR3_UINT32 line
The line number in the input stream where this token was derived from.
void * lineStart
The offset into the input stream that the line in which this token resides starts.
void(* setChannel )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)
Pointer to a function that sets the channel that this token should belong to.
void(* setCharPositionInLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)
Pointer to a function that sets the offset in the line where this token exists.
void(* setLine )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)
Pointer to a function that sets the 'line' number where this token reside.
void(* setStartIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the start index in the input stream for this token.
void(* setStopIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
Pointer to a function that sets the stop index in the input stream for this token.
void(* setText )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)
Pointer to a function that 'might' be able to set the text associated with a token.
void(* setText8 )(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)
Pointer to a function that 'might' be able to set the text associated with a token.
void(* setTokenIndex )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)
Pointer to a function that can set the token index of this token in the token input stream.
void(* setType )(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)
Pointer to a function that sets the type of this token.
ANTLR3_MARKER start
The character offset in the input stream where the text for this token starts.
ANTLR3_MARKER stop
The character offset in the input stream where the text for this token stops.
pANTLR3_STRING_FACTORY strFactory
A string factory that we can use if we ever need the text of a token and need to manufacture a pANTLR3_STRING.
ANTLR3_UINT32 textState
Indicates whether this token is carrying:.
union {
pANTLR3_UCHAR chars
Pointer that is used when the token just has a pointer to a char *, such as when a rewrite of an imaginary token supplies a string in the grammar.
pANTLR3_STRING text
Some token types actually do carry around their associated text, hence (*getText)() will return this pointer if it is not NULL.
} tokText
pANTLR3_STRING(* toString )(struct ANTLR3_COMMON_TOKEN_struct *token)
Pointer to a function that returns this token as a text representation that can be printed with embedded control codes such as
ANTLR3_UINT32 type
The actual type of this token.
ANTLR3_UINT32 user1
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user2
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
ANTLR3_UINT32 user3
Because it is a bit more of a hassle to override an ANTLR3_COMMON_TOKEN as the standard structure for a token, a number of user programmable elements are allowed in a token.
Detailed Description
Field Documentation
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::channel
ANTLR3_INT32 ANTLR3_COMMON_TOKEN_struct::charPosition
pANTLR3_UCHAR ANTLR3_COMMON_TOKEN_struct::chars
void* ANTLR3_COMMON_TOKEN_struct::custom
ANTLR3_BOOLEAN ANTLR3_COMMON_TOKEN_struct::factoryMade
void(* ANTLR3_COMMON_TOKEN_struct::freeCustom)(void *custom)
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getChannel)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_INT32(* ANTLR3_COMMON_TOKEN_struct::getCharPositionInLine)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getLine)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStartIndex)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getStopIndex)(struct ANTLR3_COMMON_TOKEN_struct *token)
pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::getText)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_MARKER(* ANTLR3_COMMON_TOKEN_struct::getTokenIndex)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_UINT32(* ANTLR3_COMMON_TOKEN_struct::getType)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::index
pANTLR3_INPUT_STREAM ANTLR3_COMMON_TOKEN_struct::input
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::line
void* ANTLR3_COMMON_TOKEN_struct::lineStart
void(* ANTLR3_COMMON_TOKEN_struct::setChannel)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 channel)
void(* ANTLR3_COMMON_TOKEN_struct::setCharPositionInLine)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_INT32 pos)
void(* ANTLR3_COMMON_TOKEN_struct::setLine)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 line)
void(* ANTLR3_COMMON_TOKEN_struct::setStartIndex)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
void(* ANTLR3_COMMON_TOKEN_struct::setStopIndex)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER index)
void(* ANTLR3_COMMON_TOKEN_struct::setText)(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_STRING text)
void(* ANTLR3_COMMON_TOKEN_struct::setText8)(struct ANTLR3_COMMON_TOKEN_struct *token, pANTLR3_UINT8 text)
void(* ANTLR3_COMMON_TOKEN_struct::setTokenIndex)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_MARKER)
void(* ANTLR3_COMMON_TOKEN_struct::setType)(struct ANTLR3_COMMON_TOKEN_struct *token, ANTLR3_UINT32 ttype)
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::start
ANTLR3_MARKER ANTLR3_COMMON_TOKEN_struct::stop
pANTLR3_STRING_FACTORY ANTLR3_COMMON_TOKEN_struct::strFactory
pANTLR3_STRING ANTLR3_COMMON_TOKEN_struct::text
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::textState
union { ... }
pANTLR3_STRING(* ANTLR3_COMMON_TOKEN_struct::toString)(struct ANTLR3_COMMON_TOKEN_struct *token)
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::type
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user1
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user2
ANTLR3_UINT32 ANTLR3_COMMON_TOKEN_struct::user3
Author