rippled
Public Member Functions | Protected Member Functions | Protected Attributes | Private Member Functions | Private Attributes | List of all members
ripple::NodeStore::Database Class Referenceabstract

Persistency layer for NodeObject. More...

Inheritance diagram for ripple::NodeStore::Database:
Inheritance graph
[legend]
Collaboration diagram for ripple::NodeStore::Database:
Collaboration graph
[legend]

Public Member Functions

 Database ()=delete
 
 Database (Scheduler &scheduler, int readThreads, Section const &config, beast::Journal j)
 Construct the node store. More...
 
virtual ~Database ()
 Destroy the node store. More...
 
virtual std::string getName () const =0
 Retrieve the name associated with this backend. More...
 
virtual void importDatabase (Database &source)=0
 Import objects from another database. More...
 
virtual std::int32_t getWriteLoad () const =0
 Retrieve the estimated number of pending write operations. More...
 
virtual void store (NodeObjectType type, Blob &&data, uint256 const &hash, std::uint32_t ledgerSeq)=0
 Store the object. More...
 
virtual bool isSameDB (std::uint32_t s1, std::uint32_t s2)=0
 
virtual void sync ()=0
 
std::shared_ptr< NodeObject > fetchNodeObject (uint256 const &hash, std::uint32_t ledgerSeq=0, FetchType fetchType=FetchType::synchronous, bool duplicate=false)
 Fetch a node object. More...
 
virtual void asyncFetch (uint256 const &hash, std::uint32_t ledgerSeq, std::function< void(std::shared_ptr< NodeObject > const &)> &&callback)
 Fetch an object without waiting. More...
 
virtual bool storeLedger (std::shared_ptr< Ledger const > const &srcLedger)=0
 Store a ledger from a different database. More...
 
virtual void sweep ()=0
 Remove expired entries from the positive and negative caches. More...
 
std::uint64_t getStoreCount () const
 Gather statistics pertaining to read and write activities. More...
 
std::uint32_t getFetchTotalCount () const
 
std::uint32_t getFetchHitCount () const
 
std::uint64_t getStoreSize () const
 
std::uint32_t getFetchSize () const
 
void getCountsJson (Json::Value &obj)
 
int fdRequired () const
 Returns the number of file descriptors the database expects to need. More...
 
virtual void stop ()
 
bool isStopping () const
 
std::uint32_t ledgersPerShard () const noexcept
 
std::uint32_t earliestLedgerSeq () const noexcept
 
std::uint32_t earliestShardIndex () const noexcept
 
std::uint32_t firstLedgerSeq (std::uint32_t shardIndex) const noexcept
 Calculates the first ledger sequence for a given shard index. More...
 
std::uint32_t lastLedgerSeq (std::uint32_t shardIndex) const noexcept
 Calculates the last ledger sequence for a given shard index. More...
 
std::uint32_t seqToShardIndex (std::uint32_t ledgerSeq) const noexcept
 Calculates the shard index for a given ledger sequence. More...
 
std::uint32_t maxLedgers (std::uint32_t shardIndex) const noexcept
 Calculates the maximum ledgers for a given shard index. More...
 

Protected Member Functions

void storeStats (std::uint64_t count, std::uint64_t sz)
 
void importInternal (Backend &dstBackend, Database &srcDB)
 
bool storeLedger (Ledger const &srcLedger, std::shared_ptr< Backend > dstBackend)
 
void updateFetchMetrics (uint64_t fetches, uint64_t hits, uint64_t duration)
 

Protected Attributes

const beast::Journal j_
 
Scheduler & scheduler_
 
int fdRequired_ {0}
 
std::atomic< std::uint32_t > fetchHitCount_ {0}
 
std::atomic< std::uint32_t > fetchSz_ {0}
 
const std::uint32_t ledgersPerShard_
 
const std::uint32_t earliestLedgerSeq_
 
const std::uint32_t earliestShardIndex_
 
const int requestBundle_
 

Private Member Functions

virtual std::shared_ptr< NodeObject > fetchNodeObject (uint256 const &hash, std::uint32_t ledgerSeq, FetchReport &fetchReport, bool duplicate)=0
 
virtual void for_each (std::function< void(std::shared_ptr< NodeObject >)> f)=0
 Visit every object in the database This is usually called during import. More...
 
virtual std::optional< Backend::Counters< std::uint64_t > > getCounters () const
 Retrieve backend read and write stats. More...
 
void threadEntry ()
 

Private Attributes

std::atomic< std::uint64_t > storeCount_ {0}
 
std::atomic< std::uint64_t > storeSz_ {0}
 
std::atomic< std::uint64_t > fetchTotalCount_ {0}
 
std::atomic< std::uint64_t > fetchDurationUs_ {0}
 
std::atomic< std::uint64_t > storeDurationUs_ {0}
 
std::mutex readLock_
 
std::condition_variable readCondVar_
 
std::map< uint256, std::vector< std::pair< std::uint32_t, std::function< void(std::shared_ptr< NodeObject > const &)> > > > read_
 
std::atomic< bool > readStopping_ = false
 
std::atomic< int > readThreads_ = 0
 
std::atomic< int > runningThreads_ = 0
 

Detailed Description

Persistency layer for NodeObject.

A Node is a ledger object which is uniquely identified by a key, which is the 256-bit hash of the body of the node. The payload is a variable length block of serialized data.

All ledger data is stored as node objects and as such, needs to be persisted between launches. Furthermore, since the set of node objects will in general be larger than the amount of available memory, purged node objects which are later accessed must be retrieved from the node store.

See also
NodeObject

Definition at line 51 of file Database.h.

Constructor & Destructor Documentation

◆ Database() [1/2]

ripple::NodeStore::Database::Database ( )
delete

◆ Database() [2/2]

ripple::NodeStore::Database::Database ( Scheduler &  scheduler,
int  readThreads,
Section const &  config,
beast::Journal  j 
)

Construct the node store.

Parameters
schedulerThe scheduler to use for performing asynchronous tasks.
readThreadsThe number of asynchronous read threads to create.
configThe configuration settings
journalDestination for logging output.

Definition at line 32 of file Database.cpp.

◆ ~Database()

ripple::NodeStore::Database::~Database ( )
virtual

Destroy the node store.

All pending operations are completed, pending writes flushed, and files closed before this returns.

Definition at line 134 of file Database.cpp.

Member Function Documentation

◆ getName()

virtual std::string ripple::NodeStore::Database::getName ( ) const
pure virtual

Retrieve the name associated with this backend.

This is used for diagnostics and may not reflect the actual path or paths used by the underlying backend.

Implemented in ripple::NodeStore::DatabaseShardImp, ripple::NodeStore::DatabaseNodeImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ importDatabase()

virtual void ripple::NodeStore::Database::importDatabase ( Database &  source)
pure virtual

Import objects from another database.

Implemented in ripple::NodeStore::DatabaseShardImp, ripple::NodeStore::DatabaseNodeImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ getWriteLoad()

virtual std::int32_t ripple::NodeStore::Database::getWriteLoad ( ) const
pure virtual

Retrieve the estimated number of pending write operations.

This is used for diagnostics.

Implemented in ripple::NodeStore::DatabaseShardImp, ripple::NodeStore::DatabaseNodeImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ store()

virtual void ripple::NodeStore::Database::store ( NodeObjectType  type,
Blob &&  data,
uint256 const &  hash,
std::uint32_t  ledgerSeq 
)
pure virtual

Store the object.

The caller's Blob parameter is overwritten.

Parameters
typeThe type of object.
dataThe payload of the object. The caller's variable is overwritten.
hashThe 256-bit hash of the payload data.
ledgerSeqThe sequence of the ledger the object belongs to.
Returns
true if the object was stored?

Implemented in ripple::NodeStore::DatabaseNodeImp, ripple::NodeStore::DatabaseRotatingImp, and ripple::NodeStore::DatabaseShardImp.

◆ isSameDB()

virtual bool ripple::NodeStore::Database::isSameDB ( std::uint32_t  s1,
std::uint32_t  s2 
)
pure virtual

Implemented in ripple::NodeStore::DatabaseNodeImp, ripple::NodeStore::DatabaseRotatingImp, and ripple::NodeStore::DatabaseShardImp.

◆ sync()

virtual void ripple::NodeStore::Database::sync ( )
pure virtual

Implemented in ripple::NodeStore::DatabaseShardImp, ripple::NodeStore::DatabaseNodeImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ fetchNodeObject() [1/2]

std::shared_ptr< NodeObject > ripple::NodeStore::Database::fetchNodeObject ( uint256 const &  hash,
std::uint32_t  ledgerSeq = 0,
FetchType  fetchType = FetchType::synchronous,
bool  duplicate = false 
)

Fetch a node object.

If the object is known to be not in the database, isn't found in the database during the fetch, or failed to load correctly during the fetch, nullptr is returned.

Note
This can be called concurrently.
Parameters
hashThe key of the object to retrieve.
ledgerSeqThe sequence of the ledger where the object is stored.
fetchTypethe type of fetch, synchronous or asynchronous.
Returns
The object, or nullptr if it couldn't be retrieved.

Definition at line 252 of file Database.cpp.

◆ asyncFetch()

void ripple::NodeStore::Database::asyncFetch ( uint256 const &  hash,
std::uint32_t  ledgerSeq,
std::function< void(std::shared_ptr< NodeObject > const &)> &&  callback 
)
virtual

Fetch an object without waiting.

If I/O is required to determine whether or not the object is present, false is returned. Otherwise, true is returned and object is set to refer to the object, or nullptr if the object is not present. If I/O is required, the I/O is scheduled and true is returned

Note
This can be called concurrently.
Parameters
hashThe key of the object to retrieve
ledgerSeqThe sequence of the ledger where the object is stored, used by the shard store.
callbackCallback function when read completes

Reimplemented in ripple::NodeStore::DatabaseNodeImp.

Definition at line 198 of file Database.cpp.

◆ storeLedger() [1/2]

virtual bool ripple::NodeStore::Database::storeLedger ( std::shared_ptr< Ledger const > const &  srcLedger)
pure virtual

Store a ledger from a different database.

Parameters
srcLedgerThe ledger to store.
Returns
true if the operation was successful

Implemented in ripple::NodeStore::DatabaseNodeImp, ripple::NodeStore::DatabaseShardImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ sweep()

virtual void ripple::NodeStore::Database::sweep ( )
pure virtual

Remove expired entries from the positive and negative caches.

Implemented in ripple::NodeStore::DatabaseNodeImp, ripple::NodeStore::DatabaseShardImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ getStoreCount()

std::uint64_t ripple::NodeStore::Database::getStoreCount ( ) const

Gather statistics pertaining to read and write activities.

Parameters
objJson object reference into which to place counters.

Definition at line 182 of file Database.h.

◆ getFetchTotalCount()

std::uint32_t ripple::NodeStore::Database::getFetchTotalCount ( ) const

Definition at line 188 of file Database.h.

◆ getFetchHitCount()

std::uint32_t ripple::NodeStore::Database::getFetchHitCount ( ) const

Definition at line 194 of file Database.h.

◆ getStoreSize()

std::uint64_t ripple::NodeStore::Database::getStoreSize ( ) const

Definition at line 200 of file Database.h.

◆ getFetchSize()

std::uint32_t ripple::NodeStore::Database::getFetchSize ( ) const

Definition at line 206 of file Database.h.

◆ getCountsJson()

void ripple::NodeStore::Database::getCountsJson ( Json::Value &  obj)

Definition at line 378 of file Database.cpp.

◆ fdRequired()

int ripple::NodeStore::Database::fdRequired ( ) const

Returns the number of file descriptors the database expects to need.

Definition at line 216 of file Database.h.

◆ stop()

void ripple::NodeStore::Database::stop ( )
virtual

Reimplemented in ripple::NodeStore::DatabaseShardImp.

Definition at line 165 of file Database.cpp.

◆ isStopping()

bool ripple::NodeStore::Database::isStopping ( ) const

Definition at line 146 of file Database.cpp.

◆ ledgersPerShard()

std::uint32_t ripple::NodeStore::Database::ledgersPerShard ( ) const
noexcept
Returns
The maximum number of ledgers stored in a shard

Definition at line 230 of file Database.h.

◆ earliestLedgerSeq()

std::uint32_t ripple::NodeStore::Database::earliestLedgerSeq ( ) const
noexcept
Returns
The earliest ledger sequence allowed

Definition at line 238 of file Database.h.

◆ earliestShardIndex()

std::uint32_t ripple::NodeStore::Database::earliestShardIndex ( ) const
noexcept
Returns
The earliest shard index

Definition at line 246 of file Database.h.

◆ firstLedgerSeq()

std::uint32_t ripple::NodeStore::Database::firstLedgerSeq ( std::uint32_t  shardIndex) const
noexcept

Calculates the first ledger sequence for a given shard index.

Parameters
shardIndexThe shard index considered
Returns
The first ledger sequence pertaining to the shard index

Definition at line 257 of file Database.h.

◆ lastLedgerSeq()

std::uint32_t ripple::NodeStore::Database::lastLedgerSeq ( std::uint32_t  shardIndex) const
noexcept

Calculates the last ledger sequence for a given shard index.

Parameters
shardIndexThe shard index considered
Returns
The last ledger sequence pertaining to the shard index

Definition at line 271 of file Database.h.

◆ seqToShardIndex()

std::uint32_t ripple::NodeStore::Database::seqToShardIndex ( std::uint32_t  ledgerSeq) const
noexcept

Calculates the shard index for a given ledger sequence.

Parameters
ledgerSeqledger sequence
Returns
The shard index of the ledger sequence

Definition at line 283 of file Database.h.

◆ maxLedgers()

std::uint32_t ripple::NodeStore::Database::maxLedgers ( std::uint32_t  shardIndex) const
noexcept

Calculates the maximum ledgers for a given shard index.

Parameters
shardIndexThe shard index considered
Returns
The maximum ledgers pertaining to the shard index
Note
The earliest shard may store less if the earliest ledger sequence truncates its beginning

Definition at line 152 of file Database.cpp.

◆ storeStats()

void ripple::NodeStore::Database::storeStats ( std::uint64_t  count,
std::uint64_t  sz 
)
protected

Definition at line 333 of file Database.h.

◆ importInternal()

void ripple::NodeStore::Database::importInternal ( Backend &  dstBackend,
Database &  srcDB 
)
protected

Definition at line 213 of file Database.cpp.

◆ storeLedger() [2/2]

bool ripple::NodeStore::Database::storeLedger ( Ledger const &  srcLedger,
std::shared_ptr< Backend >  dstBackend 
)
protected

Definition at line 279 of file Database.cpp.

◆ updateFetchMetrics()

void ripple::NodeStore::Database::updateFetchMetrics ( uint64_t  fetches,
uint64_t  hits,
uint64_t  duration 
)
protected

Definition at line 349 of file Database.h.

◆ fetchNodeObject() [2/2]

virtual std::shared_ptr<NodeObject> ripple::NodeStore::Database::fetchNodeObject ( uint256 const &  hash,
std::uint32_t  ledgerSeq,
FetchReport &  fetchReport,
bool  duplicate 
)
privatepure virtual

Implemented in ripple::NodeStore::DatabaseNodeImp, ripple::NodeStore::DatabaseRotatingImp, and ripple::NodeStore::DatabaseShardImp.

◆ for_each()

virtual void ripple::NodeStore::Database::for_each ( std::function< void(std::shared_ptr< NodeObject >)>  f)
privatepure virtual

Visit every object in the database This is usually called during import.

Note
This routine will not be called concurrently with itself or other methods.
See also
import

Implemented in ripple::NodeStore::DatabaseShardImp, ripple::NodeStore::DatabaseNodeImp, and ripple::NodeStore::DatabaseRotatingImp.

◆ getCounters()

virtual std::optional<Backend::Counters<std::uint64_t> > ripple::NodeStore::Database::getCounters ( ) const
privatevirtual

Retrieve backend read and write stats.

Note
The Counters struct is specific to and only used by CassandraBackend.

Reimplemented in ripple::NodeStore::DatabaseNodeImp.

Definition at line 401 of file Database.h.

◆ threadEntry()

void ripple::NodeStore::Database::threadEntry ( )
private

Member Data Documentation

◆ j_

const beast::Journal ripple::NodeStore::Database::j_
protected

Definition at line 301 of file Database.h.

◆ scheduler_

Scheduler& ripple::NodeStore::Database::scheduler_
protected

Definition at line 302 of file Database.h.

◆ fdRequired_

int ripple::NodeStore::Database::fdRequired_ {0}
protected

Definition at line 303 of file Database.h.

◆ fetchHitCount_

std::atomic<std::uint32_t> ripple::NodeStore::Database::fetchHitCount_ {0}
protected

Definition at line 305 of file Database.h.

◆ fetchSz_

std::atomic<std::uint32_t> ripple::NodeStore::Database::fetchSz_ {0}
protected

Definition at line 306 of file Database.h.

◆ ledgersPerShard_

const std::uint32_t ripple::NodeStore::Database::ledgersPerShard_
protected

Definition at line 314 of file Database.h.

◆ earliestLedgerSeq_

const std::uint32_t ripple::NodeStore::Database::earliestLedgerSeq_
protected

Definition at line 322 of file Database.h.

◆ earliestShardIndex_

const std::uint32_t ripple::NodeStore::Database::earliestShardIndex_
protected

Definition at line 325 of file Database.h.

◆ requestBundle_

const int ripple::NodeStore::Database::requestBundle_
protected

Definition at line 330 of file Database.h.

◆ storeCount_

std::atomic<std::uint64_t> ripple::NodeStore::Database::storeCount_ {0}
private

Definition at line 357 of file Database.h.

◆ storeSz_

std::atomic<std::uint64_t> ripple::NodeStore::Database::storeSz_ {0}
private

Definition at line 358 of file Database.h.

◆ fetchTotalCount_

std::atomic<std::uint64_t> ripple::NodeStore::Database::fetchTotalCount_ {0}
private

Definition at line 359 of file Database.h.

◆ fetchDurationUs_

std::atomic<std::uint64_t> ripple::NodeStore::Database::fetchDurationUs_ {0}
private

Definition at line 360 of file Database.h.

◆ storeDurationUs_

std::atomic<std::uint64_t> ripple::NodeStore::Database::storeDurationUs_ {0}
private

Definition at line 361 of file Database.h.

◆ readLock_

std::mutex ripple::NodeStore::Database::readLock_
mutableprivate

Definition at line 363 of file Database.h.

◆ readCondVar_

std::condition_variable ripple::NodeStore::Database::readCondVar_
private

Definition at line 364 of file Database.h.

◆ read_

std::map< uint256, std::vector<std::pair< std::uint32_t, std::function<void(std::shared_ptr<NodeObject> const&)> > > > ripple::NodeStore::Database::read_
private

Definition at line 372 of file Database.h.

◆ readStopping_

std::atomic<bool> ripple::NodeStore::Database::readStopping_ = false
private

Definition at line 374 of file Database.h.

◆ readThreads_

std::atomic<int> ripple::NodeStore::Database::readThreads_ = 0
private

Definition at line 375 of file Database.h.

◆ runningThreads_

std::atomic<int> ripple::NodeStore::Database::runningThreads_ = 0
private

Definition at line 376 of file Database.h.

Generated by   doxygen 1.8.17