0.9.8.10
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Public Member Functions | Private Member Functions | Private Attributes | List of all members
Hypertable::ClusterDefinitionFile::Tokenizer Class Reference

Splits cluster definition file into tokens. More...

#include <Tokenizer.h>

Collaboration diagram for Hypertable::ClusterDefinitionFile::Tokenizer:
Collaboration graph
[legend]

Public Member Functions

 Tokenizer (const string &fname)
 Constructor. More...
 
 Tokenizer (const string &fname, const string &content)
 Constructor. More...
 
string dirname ()
 Returns the directory path containing the cluster definition file. More...
 
bool next (Token &token)
 Returns the next token from the cluster definition file. More...
 

Private Member Functions

int identify_line_type (const char *base, const char *end)
 Identifies token type of line starting at base. More...
 
bool accumulate (const char **basep, const char *end, int type, Token &token)
 Accumulates the next token. More...
 

Private Attributes

string m_fname
 Pathname of cluster definition file. More...
 
string m_content
 Content of cluster definition file. More...
 
const char * m_next
 Pointer to beginning of next token to read. More...
 
size_t m_line {}
 Line number of end of last token read. More...
 

Detailed Description

Splits cluster definition file into tokens.

Definition at line 43 of file Tokenizer.h.

Constructor & Destructor Documentation

Tokenizer::Tokenizer ( const string &  fname)

Constructor.

Initializes m_fname with fname, reads the contents of m_fname into m_content, and then sets m_next pointing to the beginning of the content.

Parameters
fnamePathname of cluster definition file

Definition at line 54 of file Tokenizer.cc.

Tokenizer::Tokenizer ( const string &  fname,
const string &  content 
)

Constructor.

Initializes m_fname with fname, sets m_content to content, and then sets m_next pointing to the beginning of the content.

Parameters
fnamePathname of cluster definition file
contentContent of cluster definition file

Definition at line 61 of file Tokenizer.cc.

Member Function Documentation

bool Tokenizer::accumulate ( const char **  basep,
const char *  end,
int  type,
Token token 
)
private

Accumulates the next token.

This function populates token with the next token to be returned. If token is already populated, it first check to see if the following token (identified by basep, end, and type) should be merged with the existing token and if so, it merges the token text with the existing token and advances *basep to end+1, otherwise it leaves *basep unmodified and returns. Tokens that get merged are Token::COMMENT followed by Token::TASK, Token::ROLE followed by Token::CODE, and any a adjacent code tokens (e.g. Token::CODE, Token::FUNCTION, Token::BLANKLINE, Token::CONTROLFLOW). If token is populated with the token text starting at *basep because it was merged with an existing token or the existing token was empty, then it sets m_next to end.

Parameters
basepAddress of pointer to next token
endPointer to end of next token (either newline or EOF)
typeType of next token (see Token::Type)
tokenReference to token object to hold next token
Returns
true if token is ready to be returned

Definition at line 251 of file Tokenizer.cc.

string Tokenizer::dirname ( )

Returns the directory path containing the cluster definition file.

Returns the path of m_fname up to the last '/' character, or "." if m_fname does not contain any '/' character.

Returns
The directory path containing the cluster definition file.

Definition at line 66 of file Tokenizer.cc.

int Tokenizer::identify_line_type ( const char *  base,
const char *  end 
)
private

Identifies token type of line starting at base.

This function determines the token type (Token::Type) of the line starting at base and ending at end.

Parameters
basePointer to beginning of next line
endPointer to end of line or end of content
Returns
Token type of line pointed to by base
See also
Token::Type

Definition at line 190 of file Tokenizer.cc.

bool Tokenizer::next ( Token token)

Returns the next token from the cluster definition file.

This reads the next token from the cluster definition file starting at m_next. It populates token with the token text, type identifier, line number of the start of the token, and creates a translator for it with a call to Token::create_translator(). The m_next and m_line are advanced to the beginning of the next token (or EOF) and the last line of the returned token, respectively.

Parameters
tokenNext token filled in by this function
Returns
true token was read, false on EOF.

Definition at line 73 of file Tokenizer.cc.

Member Data Documentation

string Hypertable::ClusterDefinitionFile::Tokenizer::m_content
private

Content of cluster definition file.

Definition at line 115 of file Tokenizer.h.

string Hypertable::ClusterDefinitionFile::Tokenizer::m_fname
private

Pathname of cluster definition file.

Definition at line 113 of file Tokenizer.h.

size_t Hypertable::ClusterDefinitionFile::Tokenizer::m_line {}
private

Line number of end of last token read.

Definition at line 119 of file Tokenizer.h.

const char* Hypertable::ClusterDefinitionFile::Tokenizer::m_next
private

Pointer to beginning of next token to read.

Definition at line 117 of file Tokenizer.h.


The documentation for this class was generated from the following files: