Abstract base class for persistent cell lists (ones that are stored on disk). More...

#include <CellStore.h>

Inheritance diagram for Hypertable::CellStore:

Collaboration diagram for Hypertable::CellStore:

Classes
class	IndexMemoryStats

Public Member Functions
	CellStore ()

virtual	~CellStore ()

virtual void	add (const Key &key, const ByteString value)=0
	Inserts a key/value pair into the cell list. More...

virtual void	split_row_estimate_data (SplitRowDataMapT &split_row_data)
	Populates `split_row_data` with unique row and count estimates from block index. More...

virtual void	populate_index_pseudo_table_scanner (CellListScannerBuffer *scanner)
	Populates `scanner` with key/value pairs generated from CellStore index. More...

virtual int64_t	get_total_entries ()=0
	Returns the number of key/value pairs in the cell store. More...

virtual CellListScannerPtr	create_scanner (ScanContext *scan_ctx)
	Creates a scanner on this cell list. More...

virtual void	create (const char fname, size_t max_entries, PropertiesPtr &props, const TableIdentifier table_id=0)=0
	Creates a new cell store. More...

virtual void	finalize (TableIdentifier *table_identifier)=0
	Finalizes the creation of a cell store, by writing block index and metadata trailer. More...

virtual void	open (const String &fname, const String &start_row, const String &end_row, int32_t fd, int64_t file_length, CellStoreTrailer *trailer)=0
	Opens a cell store with possibly a restricted view. More...

virtual void	rescope (const String &start_row, const String &end_row)

virtual int64_t	get_blocksize ()=0
	Returns the block size used for this cell store. More...

virtual bool	may_contain (ScanContext *scan_ctx)=0
	Bloom filter lookup. More...

virtual uint64_t	disk_usage ()=0
	Returns the disk used by this cell store. More...

virtual size_t	block_count ()
	Returns the number of CellStore blocks covered by this object. More...

virtual float	compression_ratio ()=0
	Returns block compression ratio of this cell store. More...

virtual std::string &	get_filename ()=0
	Pathname of cell store file. More...

virtual int	get_file_id ()=0
	Returns a unique identifier which identifies the underlying file for caching purposes. More...

virtual CellStoreTrailer *	get_trailer ()=0
	Return a pointer to the trailer object for this cell store. More...

virtual BlockCompressionCodec *	create_block_compression_codec ()=0
	Creates a block compression codec suitable for decompressing the cell store's blocks. More...

virtual KeyDecompressor *	create_key_decompressor ()
	Creates a key decompressor suitable for decompressing the keys stored in this cell store. More...

virtual void	set_replaced_files (const std::vector< String > &old_files)
	Sets the cell store files replaced by this CellStore. More...

virtual const std::vector < String > &	get_replaced_files ()
	Returns all the cell store files replaced by this CellStore. More...

virtual void	display_block_info ()=0
	Displays block information to stdout. More...

virtual size_t	bloom_filter_size ()=0
	Return Bloom filter size. More...

virtual int32_t	get_fd ()=0
	Returns the open file descriptor for the CellStore file. More...

virtual int32_t	reopen_fd ()=0
	Closes and reopens the underlying CellStore file. More...

virtual int64_t	bloom_filter_memory_used ()=0
	Returns the amount of memory consumed by the bloom filter. More...

virtual int64_t	block_index_memory_used ()=0
	Returns the amount of memory consumed by the block index. More...

virtual int64_t	end_of_last_block ()=0
	Returns the offset of the end of the last block in the cell store. More...

virtual uint64_t	purge_indexes ()=0
	Purges bloom filter and block indexes. More...

virtual void	get_index_memory_stats (IndexMemoryStats *statsp)
	Returns amount of purgeable index memory available. More...

virtual bool	restricted_range ()=0
	Returns true if the cellstore was opened with a restricted range. More...

uint64_t	bytes_read ()
	Returns the number of "uncompressed" bytes read from the underlying filesystem. More...

void	decrement_index_refcount ()
	Decrement index reference count. More...

virtual uint16_t	block_header_format ()=0

Public Member Functions inherited from Hypertable::CellList
virtual	~CellList ()

virtual const char *	get_start_row ()
	Returns the start row of this cell list. More...

virtual const char *	get_end_row ()
	Returns the end row of this cell list. More...

Static Public Attributes
static const char	DATA_BLOCK_MAGIC [10]

static const char	INDEX_FIXED_BLOCK_MAGIC [10]

static const char	INDEX_VARIABLE_BLOCK_MAGIC [10]

Protected Attributes
std::mutex	m_mutex

IndexMemoryStats	m_index_stats

std::vector< String >	m_replaced_files

uint64_t	m_bytes_read

size_t	m_block_count

uint32_t	m_index_refcount

Protected Attributes inherited from Hypertable::CellList
std::string	m_start_row

std::string	m_end_row

Additional Inherited Members
Public Types inherited from Hypertable::CellList
typedef std::pair< const char *, int64_t >	SplitRowDataValue

typedef StlAllocator < SplitRowDataValue >	SplitRowDataAlloc

typedef std::map< const char *, int64_t, LtCstr, SplitRowDataAlloc >	SplitRowDataMapT

Detailed Description

Abstract base class for persistent cell lists (ones that are stored on disk).

Definition at line 57 of file CellStore.h.

Constructor & Destructor Documentation

Hypertable::CellStore::CellStore ( )

inline

Definition at line 71 of file CellStore.h.

virtual Hypertable::CellStore::~CellStore ( )

inlinevirtual

Definition at line 73 of file CellStore.h.

Member Function Documentation

virtual void Hypertable::CellStore::add	(	const Key &	key,
		const ByteString	value
	)

pure virtual

Inserts a key/value pair into the cell list.

Parameters

key	key object
value	ByteString representing value

Implements Hypertable::CellList.

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV2, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual size_t Hypertable::CellStore::block_count ( )

inlinevirtual

Returns the number of CellStore blocks covered by this object.

Returns: block count

Definition at line 174 of file CellStore.h.

virtual uint16_t Hypertable::CellStore::block_header_format ( )

pure virtual

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual int64_t Hypertable::CellStore::block_index_memory_used ( )

pure virtual

Returns the amount of memory consumed by the block index.

Returns: memory used by block index

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual int64_t Hypertable::CellStore::bloom_filter_memory_used ( )

pure virtual

Returns the amount of memory consumed by the bloom filter.

Returns: memory used by bloom filter

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual size_t Hypertable::CellStore::bloom_filter_size ( )

pure virtual

Return Bloom filter size.

Returns: size of bloom filter

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

uint64_t Hypertable::CellStore::bytes_read ( )

inline

Returns the number of "uncompressed" bytes read from the underlying filesystem.

Returns: number of uncompressed bytes read from filesystem

Definition at line 311 of file CellStore.h.

virtual float Hypertable::CellStore::compression_ratio ( )

pure virtual

Returns block compression ratio of this cell store.

Returns: compression ratio

Implemented in Hypertable::CellStoreV3, Hypertable::CellStoreV6, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV7, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::create	(	const char *	fname,
		size_t	max_entries,
		PropertiesPtr &	props,
		const TableIdentifier *	table_id = `0`
	)

pure virtual

Creates a new cell store.

Parameters

fname	name of file to contain the cell store
max_entries	maximum number of entries the cell store is expected to have
props	cellstore specific properties
table_id	Table identifier

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV1, Hypertable::CellStoreV2, and Hypertable::CellStoreV0.

virtual BlockCompressionCodec* Hypertable::CellStore::create_block_compression_codec ( )

pure virtual

Creates a block compression codec suitable for decompressing the cell store's blocks.

Returns: pointer to compression codec

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

KeyDecompressor * CellStore::create_key_decompressor ( )

virtual

Creates a key decompressor suitable for decompressing the keys stored in this cell store.

Returns: pointer to key decompressor

Reimplemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, and Hypertable::CellStoreV5.

Definition at line 42 of file CellStore.cc.

virtual CellListScannerPtr Hypertable::CellStore::create_scanner ( ScanContext * scan_ctx )

inlinevirtual

Creates a scanner on this cell list.

Parameters

scan_ctx smart pointer to scan context

Returns: pointer to newly allocated scanner

Reimplemented from Hypertable::CellList.

Reimplemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

Definition at line 99 of file CellStore.h.

void Hypertable::CellStore::decrement_index_refcount ( )

inline

Decrement index reference count.

Definition at line 318 of file CellStore.h.

virtual uint64_t Hypertable::CellStore::disk_usage ( )

pure virtual

Returns the disk used by this cell store.

If the cell store is opened with a restricted range, then it returns an estimate of the disk used by that range.

Returns: disk used by this cell store or portion thereof

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV3, Hypertable::CellStoreV7, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV5, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::display_block_info ( )

pure virtual

Displays block information to stdout.

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual int64_t Hypertable::CellStore::end_of_last_block ( )

pure virtual

Returns the offset of the end of the last block in the cell store.

Returns: offset of end of last block

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::finalize ( TableIdentifier * table_identifier )

pure virtual

Finalizes the creation of a cell store, by writing block index and metadata trailer.

Parameters

table_identifier table identifier

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV2, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual int64_t Hypertable::CellStore::get_blocksize ( )

pure virtual

Returns the block size used for this cell store.

The block size is the amount of uncompressed key/value pairs to collect before compressing and storing as a compressed block in the cell store.

Returns: block size

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV2, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual int32_t Hypertable::CellStore::get_fd ( )

pure virtual

Returns the open file descriptor for the CellStore file.

Returns: open file descriptor

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual int Hypertable::CellStore::get_file_id ( )

pure virtual

Returns a unique identifier which identifies the underlying file for caching purposes.

Returns: unique file id

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual std::string& Hypertable::CellStore::get_filename ( )

pure virtual

Pathname of cell store file.

Returns: string reference to the cell store path name

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::get_index_memory_stats ( IndexMemoryStats * statsp )

inlinevirtual

Returns amount of purgeable index memory available.

Definition at line 293 of file CellStore.h.

const std::vector< String > & CellStore::get_replaced_files ( )

virtual

Returns all the cell store files replaced by this CellStore.

Returns: vector of strings with names of the files that this cell store replaces

Reimplemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, and Hypertable::CellStoreV5.

Definition at line 50 of file CellStore.cc.

virtual int64_t Hypertable::CellStore::get_total_entries ( )

pure virtual

Returns the number of key/value pairs in the cell store.

Returns: Number of key/value pairs in list

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual CellStoreTrailer* Hypertable::CellStore::get_trailer ( )

pure virtual

Return a pointer to the trailer object for this cell store.

Returns: pointer to the internal trailer of this cell store

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual bool Hypertable::CellStore::may_contain ( ScanContext * scan_ctx )

pure virtual

Bloom filter lookup.

Parameters

scan_ctx Scan context

Returns: true if cell store may contain row referenced by scan_ctx

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV2, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::open	(	const String &	fname,
		const String &	start_row,
		const String &	end_row,
		int32_t	fd,
		int64_t	file_length,
		CellStoreTrailer *	trailer
	)

pure virtual

Opens a cell store with possibly a restricted view.

When a range splits, the cell stores that comprise the range get shared between the two newly created ranges. This method allows each of the two ranges to open the same cell store but view different portions of it.

Parameters

fname	pathname of file containing cell store
start_row	restricts view of this store to key/value pairs that are greater than this value
end_row	restricts view of this store to key/value pairs that are less than or equal to this value
fd	cell store file descriptor
file_length	length of cell store file
trailer	pointer to trailer object

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV5, Hypertable::CellStoreV2, Hypertable::CellStoreV1, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::populate_index_pseudo_table_scanner ( CellListScannerBuffer * scanner )

inlinevirtual

Populates scanner with key/value pairs generated from CellStore index.

Parameters

scanner Pointer to CellListScannerBuffer to receive key/value pairs

Reimplemented in Hypertable::CellStoreV6, and Hypertable::CellStoreV7.

Definition at line 92 of file CellStore.h.

virtual uint64_t Hypertable::CellStore::purge_indexes ( )

pure virtual

Purges bloom filter and block indexes.

Returns: amount of memory purged

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual int32_t Hypertable::CellStore::reopen_fd ( )

pure virtual

Closes and reopens the underlying CellStore file.

Returns: new file descriptor

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

virtual void Hypertable::CellStore::rescope	(	const String &	start_row,
		const String &	end_row
	)

inlinevirtual

Reimplemented in Hypertable::CellStoreV6, and Hypertable::CellStoreV7.

Definition at line 140 of file CellStore.h.

virtual bool Hypertable::CellStore::restricted_range ( )

pure virtual

Returns true if the cellstore was opened with a restricted range.

Returns: true if cellstore opened with restricted range

Implemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, Hypertable::CellStoreV3, Hypertable::CellStoreV4, Hypertable::CellStoreV2, Hypertable::CellStoreV1, Hypertable::CellStoreV5, and Hypertable::CellStoreV0.

void CellStore::set_replaced_files ( const std::vector< String > & old_files )

virtual

Sets the cell store files replaced by this CellStore.

Definition at line 46 of file CellStore.cc.

virtual void Hypertable::CellStore::split_row_estimate_data ( SplitRowDataMapT & split_row_data )

inlinevirtual

Populates split_row_data with unique row and count estimates from block index.

Parameters

split_row_data Reference to accumulator map holding unique rows and counts taken from block index.

Note: split_row_data should not be cleared

Reimplemented from Hypertable::CellList.

Reimplemented in Hypertable::CellStoreV6, Hypertable::CellStoreV7, and Hypertable::CellStoreV5.

Definition at line 83 of file CellStore.h.

Member Data Documentation

const char CellStore::DATA_BLOCK_MAGIC

static

Initial value:

=

{ 'D','a','t','a','-','-','-','-','-','-' }

Definition at line 325 of file CellStore.h.

const char CellStore::INDEX_FIXED_BLOCK_MAGIC

static

Initial value:

=

{ 'I','d','x','F','i','x','-','-','-','-' }

Definition at line 326 of file CellStore.h.

const char CellStore::INDEX_VARIABLE_BLOCK_MAGIC

static

Initial value:

=

{ 'I','d','x','V','a','r','-','-','-','-' }

Definition at line 327 of file CellStore.h.

size_t Hypertable::CellStore::m_block_count

protected

Definition at line 335 of file CellStore.h.

uint64_t Hypertable::CellStore::m_bytes_read

protected

Definition at line 334 of file CellStore.h.

uint32_t Hypertable::CellStore::m_index_refcount

protected

Definition at line 336 of file CellStore.h.

IndexMemoryStats Hypertable::CellStore::m_index_stats

protected

Definition at line 332 of file CellStore.h.

std::mutex Hypertable::CellStore::m_mutex

protected

Definition at line 331 of file CellStore.h.

std::vector<String> Hypertable::CellStore::m_replaced_files

protected

Definition at line 333 of file CellStore.h.

The documentation for this class was generated from the following files:

/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/CellStore.h
/home/doug/src/hypertable/src/cc/Hypertable/RangeServer/CellStore.cc

Classes

Public Member Functions

Static Public Attributes

Protected Attributes

Additional Inherited Members

Detailed Description

Constructor & Destructor Documentation

Member Function Documentation

Member Data Documentation