0.9.8.10
 All Classes Namespaces Files Functions Variables Typedefs Enumerations Enumerator Friends Macros Modules Pages
Classes | Public Member Functions | Protected Types | Protected Member Functions | Protected Attributes | List of all members
Hypertable::CellStoreV6 Class Reference

#include <CellStoreV6.h>

Inheritance diagram for Hypertable::CellStoreV6:
Inheritance graph
[legend]
Collaboration diagram for Hypertable::CellStoreV6:
Collaboration graph
[legend]

Classes

class  IndexBuilder
 

Public Member Functions

 CellStoreV6 (Filesystem *filesys)
 
 CellStoreV6 (Filesystem *filesys, SchemaPtr &schema)
 
virtual ~CellStoreV6 ()
 
void create (const char *fname, size_t max_entries, PropertiesPtr &props, const TableIdentifier *table_id=0) override
 Creates a new cell store. More...
 
void add (const Key &key, const ByteString value) override
 Inserts a key/value pair into the cell list. More...
 
void finalize (TableIdentifier *table_identifier) override
 Finalizes the creation of a cell store, by writing block index and metadata trailer. More...
 
void open (const String &fname, const String &start_row, const String &end_row, int32_t fd, int64_t file_length, CellStoreTrailer *trailer) override
 Opens a cell store with possibly a restricted view. More...
 
void rescope (const String &start_row, const String &end_row) override
 
int64_t get_blocksize () override
 Returns the block size used for this cell store. More...
 
bool may_contain (ScanContext *scan_ctx) override
 Bloom filter lookup. More...
 
uint64_t disk_usage () override
 Returns the disk used by this cell store. More...
 
float compression_ratio () override
 Returns block compression ratio of this cell store. More...
 
void split_row_estimate_data (SplitRowDataMapT &split_row_data) override
 Populates split_row_data with unique row and count estimates from block index. More...
 
void populate_index_pseudo_table_scanner (CellListScannerBuffer *scanner) override
 Populates scanner with key/value pairs generated from CellStore index. More...
 
int64_t get_total_entries () override
 Returns the number of key/value pairs in the cell store. More...
 
std::string & get_filename () override
 Pathname of cell store file. More...
 
int get_file_id () override
 Returns a unique identifier which identifies the underlying file for caching purposes. More...
 
CellListScannerPtr create_scanner (ScanContext *scan_ctx) override
 Creates a scanner on this cell list. More...
 
BlockCompressionCodeccreate_block_compression_codec () override
 Creates a block compression codec suitable for decompressing the cell store's blocks. More...
 
KeyDecompressorcreate_key_decompressor () override
 Creates a key decompressor suitable for decompressing the keys stored in this cell store. More...
 
void display_block_info () override
 Displays block information to stdout. More...
 
int64_t end_of_last_block () override
 Returns the offset of the end of the last block in the cell store. More...
 
size_t bloom_filter_size () override
 Return Bloom filter size. More...
 
int64_t bloom_filter_memory_used () override
 Returns the amount of memory consumed by the bloom filter. More...
 
int64_t block_index_memory_used () override
 Returns the amount of memory consumed by the block index. More...
 
uint64_t purge_indexes () override
 Purges bloom filter and block indexes. More...
 
bool restricted_range () override
 Returns true if the cellstore was opened with a restricted range. More...
 
const std::vector< String > & get_replaced_files () override
 Returns all the cell store files replaced by this CellStore. More...
 
int32_t get_fd () override
 Returns the open file descriptor for the CellStore file. More...
 
int32_t reopen_fd () override
 Closes and reopens the underlying CellStore file. More...
 
CellStoreTrailerget_trailer () override
 Return a pointer to the trailer object for this cell store. More...
 
uint16_t block_header_format () override
 
- Public Member Functions inherited from Hypertable::CellStore
 CellStore ()
 
virtual ~CellStore ()
 
virtual size_t block_count ()
 Returns the number of CellStore blocks covered by this object. More...
 
virtual void set_replaced_files (const std::vector< String > &old_files)
 Sets the cell store files replaced by this CellStore. More...
 
virtual void get_index_memory_stats (IndexMemoryStats *statsp)
 Returns amount of purgeable index memory available. More...
 
uint64_t bytes_read ()
 Returns the number of "uncompressed" bytes read from the underlying filesystem. More...
 
void decrement_index_refcount ()
 Decrement index reference count. More...
 
- Public Member Functions inherited from Hypertable::CellList
virtual ~CellList ()
 
virtual const char * get_start_row ()
 Returns the start row of this cell list. More...
 
virtual const char * get_end_row ()
 Returns the end row of this cell list. More...
 

Protected Types

typedef BlobHashSet BloomFilterItems
 

Protected Member Functions

void create_bloom_filter (bool is_approx=false)
 
void load_bloom_filter ()
 
void load_block_index ()
 
void load_replaced_files ()
 

Protected Attributes

Filesystemm_filesys
 
SchemaPtr m_schema
 
int32_t m_fd {-1}
 
std::string m_filename
 
bool m_64bit_index {}
 
CellStoreTrailerV6 m_trailer
 
BlockCompressionCodecm_compressor {}
 
DynamicBuffer m_buffer
 
IndexBuilder m_index_builder
 
DispatchHandlerSynchronizer m_sync_handler
 
uint32_t m_outstanding_appends {}
 
int64_t m_offset {}
 
int64_t m_file_length {}
 
int64_t m_disk_usage {}
 
int m_file_id {}
 
float m_uncompressed_data {}
 
float m_compressed_data {}
 
int64_t m_uncompressed_blocksize {}
 
BlockCompressionCodec::Args m_compressor_args
 
size_t m_max_entries {}
 
BloomFilterMode m_bloom_filter_mode {BLOOM_FILTER_DISABLED}
 
BloomFilterItemsm_bloom_filter_items {}
 
int64_t m_max_approx_items {}
 
float m_bloom_bits_per_item {}
 
float m_filter_false_positive_prob {}
 
KeyCompressorPtr m_key_compressor {}
 
bool m_restricted_range {}
 
int64_t * m_column_ttl {}
 
bool m_replaced_files_loaded {}
 
BloomFilterWithChecksumm_bloom_filter {}
 Bloom filter. More...
 
CellStoreBlockIndexArray
< uint32_t > 
m_index_map32
 32-bit block index More...
 
CellStoreBlockIndexArray< int64_t > m_index_map64
 64-bit block index More...
 
- Protected Attributes inherited from Hypertable::CellStore
std::mutex m_mutex
 
IndexMemoryStats m_index_stats
 
std::vector< Stringm_replaced_files
 
uint64_t m_bytes_read
 
size_t m_block_count
 
uint32_t m_index_refcount
 
- Protected Attributes inherited from Hypertable::CellList
std::string m_start_row
 
std::string m_end_row
 

Additional Inherited Members

- Public Types inherited from Hypertable::CellList
typedef std::pair< const char
*, int64_t > 
SplitRowDataValue
 
typedef StlAllocator
< SplitRowDataValue
SplitRowDataAlloc
 
typedef std::map< const char
*, int64_t, LtCstr,
SplitRowDataAlloc
SplitRowDataMapT
 
- Static Public Attributes inherited from Hypertable::CellStore
static const char DATA_BLOCK_MAGIC [10]
 
static const char INDEX_FIXED_BLOCK_MAGIC [10]
 
static const char INDEX_VARIABLE_BLOCK_MAGIC [10]
 

Detailed Description

Definition at line 65 of file CellStoreV6.h.

Member Typedef Documentation

Definition at line 164 of file CellStoreV6.h.

Constructor & Destructor Documentation

CellStoreV6::CellStoreV6 ( Filesystem filesys)

Definition at line 69 of file CellStoreV6.cc.

CellStoreV6::CellStoreV6 ( Filesystem filesys,
SchemaPtr schema 
)

Definition at line 75 of file CellStoreV6.cc.

CellStoreV6::~CellStoreV6 ( )
virtual

Definition at line 81 of file CellStoreV6.cc.

Member Function Documentation

void CellStoreV6::add ( const Key key,
const ByteString  value 
)
overridevirtual

Inserts a key/value pair into the cell list.

Parameters
keykey object
valueByteString representing value

Implements Hypertable::CellStore.

Definition at line 446 of file CellStoreV6.cc.

uint16_t CellStoreV6::block_header_format ( )
overridevirtual

Implements Hypertable::CellStore.

Definition at line 1096 of file CellStoreV6.cc.

int64_t Hypertable::CellStoreV6::block_index_memory_used ( )
inlineoverridevirtual

Returns the amount of memory consumed by the block index.

Returns
memory used by block index

Implements Hypertable::CellStore.

Definition at line 132 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::bloom_filter_memory_used ( )
inlineoverridevirtual

Returns the amount of memory consumed by the bloom filter.

Returns
memory used by bloom filter

Implements Hypertable::CellStore.

Definition at line 127 of file CellStoreV6.h.

size_t Hypertable::CellStoreV6::bloom_filter_size ( )
inlineoverridevirtual

Return Bloom filter size.

Returns
size of bloom filter

Implements Hypertable::CellStore.

Definition at line 122 of file CellStoreV6.h.

float Hypertable::CellStoreV6::compression_ratio ( )
inlineoverridevirtual

Returns block compression ratio of this cell store.

Returns
compression ratio

Implements Hypertable::CellStore.

Definition at line 99 of file CellStoreV6.h.

void CellStoreV6::create ( const char *  fname,
size_t  max_entries,
PropertiesPtr props,
const TableIdentifier table_id = 0 
)
overridevirtual

Creates a new cell store.

Parameters
fnamename of file to contain the cell store
max_entriesmaximum number of entries the cell store is expected to have
propscellstore specific properties
table_idTable identifier

Implements Hypertable::CellStore.

Definition at line 179 of file CellStoreV6.cc.

BlockCompressionCodec * CellStoreV6::create_block_compression_codec ( )
overridevirtual

Creates a block compression codec suitable for decompressing the cell store's blocks.

Returns
pointer to compression codec

Implements Hypertable::CellStore.

Definition at line 99 of file CellStoreV6.cc.

void CellStoreV6::create_bloom_filter ( bool  is_approx = false)
protected

Definition at line 274 of file CellStoreV6.cc.

KeyDecompressor * CellStoreV6::create_key_decompressor ( )
overridevirtual

Creates a key decompressor suitable for decompressing the keys stored in this cell store.

Returns
pointer to key decompressor

Reimplemented from Hypertable::CellStore.

Definition at line 104 of file CellStoreV6.cc.

CellListScannerPtr CellStoreV6::create_scanner ( ScanContext scan_ctx)
overridevirtual

Creates a scanner on this cell list.

Parameters
scan_ctxsmart pointer to scan context
Returns
pointer to newly allocated scanner

Reimplemented from Hypertable::CellStore.

Definition at line 143 of file CellStoreV6.cc.

uint64_t Hypertable::CellStoreV6::disk_usage ( )
inlineoverridevirtual

Returns the disk used by this cell store.

If the cell store is opened with a restricted range, then it returns an estimate of the disk used by that range.

Returns
disk used by this cell store or portion thereof

Implements Hypertable::CellStore.

Definition at line 98 of file CellStoreV6.h.

void CellStoreV6::display_block_info ( )
overridevirtual

Displays block information to stdout.

Implements Hypertable::CellStore.

Definition at line 1085 of file CellStoreV6.cc.

int64_t Hypertable::CellStoreV6::end_of_last_block ( )
inlineoverridevirtual

Returns the offset of the end of the last block in the cell store.

Returns
offset of end of last block

Implements Hypertable::CellStore.

Definition at line 120 of file CellStoreV6.h.

void CellStoreV6::finalize ( TableIdentifier table_identifier)
overridevirtual

Finalizes the creation of a cell store, by writing block index and metadata trailer.

Parameters
table_identifiertable identifier

Chop the Index buffers down to the exact length

Write fixed index

Write variable index

Set up index

close file for writing

Set file length

Re-open file for reading

Implements Hypertable::CellStore.

Definition at line 557 of file CellStoreV6.cc.

int64_t Hypertable::CellStoreV6::get_blocksize ( )
inlineoverridevirtual

Returns the block size used for this cell store.

The block size is the amount of uncompressed key/value pairs to collect before compressing and storing as a compressed block in the cell store.

Returns
block size

Implements Hypertable::CellStore.

Definition at line 96 of file CellStoreV6.h.

int32_t Hypertable::CellStoreV6::get_fd ( )
inlineoverridevirtual

Returns the open file descriptor for the CellStore file.

Returns
open file descriptor

Implements Hypertable::CellStore.

Definition at line 141 of file CellStoreV6.h.

int Hypertable::CellStoreV6::get_file_id ( )
inlineoverridevirtual

Returns a unique identifier which identifies the underlying file for caching purposes.

Returns
unique file id

Implements Hypertable::CellStore.

Definition at line 115 of file CellStoreV6.h.

std::string& Hypertable::CellStoreV6::get_filename ( )
inlineoverridevirtual

Pathname of cell store file.

Returns
string reference to the cell store path name

Implements Hypertable::CellStore.

Definition at line 114 of file CellStoreV6.h.

const std::vector< String > & CellStoreV6::get_replaced_files ( )
overridevirtual

Returns all the cell store files replaced by this CellStore.

Returns
vector of strings with names of the files that this cell store replaces

Reimplemented from Hypertable::CellStore.

Definition at line 305 of file CellStoreV6.cc.

int64_t Hypertable::CellStoreV6::get_total_entries ( )
inlineoverridevirtual

Returns the number of key/value pairs in the cell store.

Returns
Number of key/value pairs in list

Implements Hypertable::CellStore.

Definition at line 113 of file CellStoreV6.h.

CellStoreTrailer* Hypertable::CellStoreV6::get_trailer ( )
inlineoverridevirtual

Return a pointer to the trailer object for this cell store.

Returns
pointer to the internal trailer of this cell store

Implements Hypertable::CellStore.

Definition at line 154 of file CellStoreV6.h.

void CellStoreV6::load_block_index ( )
protected

Read index data

inflate fixed index

inflate variable index

Set up index

Definition at line 932 of file CellStoreV6.cc.

void CellStoreV6::load_bloom_filter ( )
protected

Definition at line 359 of file CellStoreV6.cc.

void CellStoreV6::load_replaced_files ( )
protected

Read index data

inflate replaced files

Definition at line 312 of file CellStoreV6.cc.

bool CellStoreV6::may_contain ( ScanContext scan_ctx)
overridevirtual

Bloom filter lookup.

Parameters
scan_ctxScan context
Returns
true if cell store may contain row referenced by scan_ctx

Implements Hypertable::CellStore.

Definition at line 1030 of file CellStoreV6.cc.

void CellStoreV6::open ( const String fname,
const String start_row,
const String end_row,
int32_t  fd,
int64_t  file_length,
CellStoreTrailer trailer 
)
overridevirtual

Opens a cell store with possibly a restricted view.

When a range splits, the cell stores that comprise the range get shared between the two newly created ranges. This method allows each of the two ranges to open the same cell store but view different portions of it.

Parameters
fnamepathname of file containing cell store
start_rowrestricts view of this store to key/value pairs that are greater than this value
end_rowrestricts view of this store to key/value pairs that are less than or equal to this value
fdcell store file descriptor
file_lengthlength of cell store file
trailerpointer to trailer object

Sanity check trailer

Implements Hypertable::CellStore.

Definition at line 862 of file CellStoreV6.cc.

void CellStoreV6::populate_index_pseudo_table_scanner ( CellListScannerBuffer scanner)
overridevirtual

Populates scanner with key/value pairs generated from CellStore index.

This method will first load the CellStore block index into memory, if it is not already loaded, and then it will call the CellStoreBlockIndexArray::populate_pseudo_table_scanner method to populate scanner with synthesized .cellstore.index pseudo-table cells.

Parameters
scannerPointer to CellListScannerBuffer to receive key/value pairs

Reimplemented from Hypertable::CellStore.

Definition at line 123 of file CellStoreV6.cc.

uint64_t CellStoreV6::purge_indexes ( )
overridevirtual

Purges bloom filter and block indexes.

Returns
amount of memory purged

Implements Hypertable::CellStore.

Definition at line 416 of file CellStoreV6.cc.

int32_t Hypertable::CellStoreV6::reopen_fd ( )
inlineoverridevirtual

Closes and reopens the underlying CellStore file.

Returns
new file descriptor

Implements Hypertable::CellStore.

Definition at line 146 of file CellStoreV6.h.

void CellStoreV6::rescope ( const String start_row,
const String end_row 
)
overridevirtual

Reimplemented from Hypertable::CellStore.

Definition at line 900 of file CellStoreV6.cc.

bool Hypertable::CellStoreV6::restricted_range ( )
inlineoverridevirtual

Returns true if the cellstore was opened with a restricted range.

Returns
true if cellstore opened with restricted range

Implements Hypertable::CellStore.

Definition at line 138 of file CellStoreV6.h.

void CellStoreV6::split_row_estimate_data ( SplitRowDataMapT split_row_data)
overridevirtual

Populates split_row_data with unique row and count estimates from block index.

Parameters
split_row_dataReference to accumulator map holding unique rows and counts taken from block index.
Note
split_row_data should not be cleared

Reimplemented from Hypertable::CellStore.

Definition at line 108 of file CellStoreV6.cc.

Member Data Documentation

bool Hypertable::CellStoreV6::m_64bit_index {}
protected

Definition at line 170 of file CellStoreV6.h.

float Hypertable::CellStoreV6::m_bloom_bits_per_item {}
protected

Definition at line 189 of file CellStoreV6.h.

BloomFilterWithChecksum* Hypertable::CellStoreV6::m_bloom_filter {}
protected

Bloom filter.

Definition at line 199 of file CellStoreV6.h.

BloomFilterItems* Hypertable::CellStoreV6::m_bloom_filter_items {}
protected

Definition at line 187 of file CellStoreV6.h.

BloomFilterMode Hypertable::CellStoreV6::m_bloom_filter_mode {BLOOM_FILTER_DISABLED}
protected

Definition at line 186 of file CellStoreV6.h.

DynamicBuffer Hypertable::CellStoreV6::m_buffer
protected

Definition at line 173 of file CellStoreV6.h.

int64_t* Hypertable::CellStoreV6::m_column_ttl {}
protected

Definition at line 193 of file CellStoreV6.h.

float Hypertable::CellStoreV6::m_compressed_data {}
protected

Definition at line 182 of file CellStoreV6.h.

BlockCompressionCodec* Hypertable::CellStoreV6::m_compressor {}
protected

Definition at line 172 of file CellStoreV6.h.

BlockCompressionCodec::Args Hypertable::CellStoreV6::m_compressor_args
protected

Definition at line 184 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::m_disk_usage {}
protected

Definition at line 179 of file CellStoreV6.h.

int32_t Hypertable::CellStoreV6::m_fd {-1}
protected

Definition at line 168 of file CellStoreV6.h.

int Hypertable::CellStoreV6::m_file_id {}
protected

Definition at line 180 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::m_file_length {}
protected

Definition at line 178 of file CellStoreV6.h.

std::string Hypertable::CellStoreV6::m_filename
protected

Definition at line 169 of file CellStoreV6.h.

Filesystem* Hypertable::CellStoreV6::m_filesys
protected

Definition at line 166 of file CellStoreV6.h.

float Hypertable::CellStoreV6::m_filter_false_positive_prob {}
protected

Definition at line 190 of file CellStoreV6.h.

IndexBuilder Hypertable::CellStoreV6::m_index_builder
protected

Definition at line 174 of file CellStoreV6.h.

CellStoreBlockIndexArray<uint32_t> Hypertable::CellStoreV6::m_index_map32
protected

32-bit block index

Definition at line 202 of file CellStoreV6.h.

CellStoreBlockIndexArray<int64_t> Hypertable::CellStoreV6::m_index_map64
protected

64-bit block index

Definition at line 205 of file CellStoreV6.h.

KeyCompressorPtr Hypertable::CellStoreV6::m_key_compressor {}
protected

Definition at line 191 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::m_max_approx_items {}
protected

Definition at line 188 of file CellStoreV6.h.

size_t Hypertable::CellStoreV6::m_max_entries {}
protected

Definition at line 185 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::m_offset {}
protected

Definition at line 177 of file CellStoreV6.h.

uint32_t Hypertable::CellStoreV6::m_outstanding_appends {}
protected

Definition at line 176 of file CellStoreV6.h.

bool Hypertable::CellStoreV6::m_replaced_files_loaded {}
protected

Definition at line 194 of file CellStoreV6.h.

bool Hypertable::CellStoreV6::m_restricted_range {}
protected

Definition at line 192 of file CellStoreV6.h.

SchemaPtr Hypertable::CellStoreV6::m_schema
protected

Definition at line 167 of file CellStoreV6.h.

DispatchHandlerSynchronizer Hypertable::CellStoreV6::m_sync_handler
protected

Definition at line 175 of file CellStoreV6.h.

CellStoreTrailerV6 Hypertable::CellStoreV6::m_trailer
protected

Definition at line 171 of file CellStoreV6.h.

int64_t Hypertable::CellStoreV6::m_uncompressed_blocksize {}
protected

Definition at line 183 of file CellStoreV6.h.

float Hypertable::CellStoreV6::m_uncompressed_data {}
protected

Definition at line 181 of file CellStoreV6.h.


The documentation for this class was generated from the following files: