rottenwheel/monero
1
0
Fork 0
mirror of https://codeberg.org/anoncontributorxmr/monero.git synced 2024-12-13 01:20:27 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
monero/src/cryptonote_core/cryptonote_core.cpp

1869 lines
74 KiB
C++
Raw Normal View History

Update 2018 copyright
2018-01-07 05:05:16 +00:00
// Copyright (c) 2014-2018, The Monero Project
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// All rights reserved.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// Redistribution and use in source and binary forms, with or without modification, are
// permitted provided that the following conditions are met:
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 1. Redistributions of source code must retain the above copyright notice, this list of
// conditions and the following disclaimer.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 2. Redistributions in binary form must reproduce the above copyright notice, this list
// of conditions and the following disclaimer in the documentation and/or other
// materials provided with the distribution.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// 3. Neither the name of the copyright holder nor the names of its contributors may be
// used to endorse or promote products derived from this software without specific
// prior written permission.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
// MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL
// THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
// PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF
// THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
//
License updated to BSD 3-clause
2014-07-23 13:03:52 +00:00
// Parts of this file are originally copyright (c) 2012-2013 The Cryptonote developers
moved all stuff to github
2014-03-03 22:07:58 +00:00
move includes around to lessen overall load
2017-11-25 22:25:05 +00:00
#include <boost/algorithm/string.hpp>
#include "string_tools.h"
moved all stuff to github
2014-03-03 22:07:58 +00:00
using namespace epee;
#include <unordered_set>
#include "cryptonote_core.h"
#include "common/util.h"
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
#include "common/updates.h"
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
#include "common/download.h"
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
#include "common/threadpool.h"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
#include "common/command_line.h"
moved all stuff to github
2014-03-03 22:07:58 +00:00
#include "warnings.h"
#include "crypto/crypto.h"
#include "cryptonote_config.h"
#include "misc_language.h"
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
#include "file_io_utils.h"
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 19:24:43 +00:00
#include <csignal>
move checkpoints in a separate library
2017-09-10 16:35:59 +00:00
#include "checkpoints/checkpoints.h"
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
#include "ringct/rctTypes.h"
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
#include "blockchain_db/blockchain_db.h"
rct: split rct checks between semantics and other Semantics can be checked early
2017-01-14 13:29:08 +00:00
#include "ringct/rctSigs.h"
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 20:18:08 +00:00
#include "common/notify.h"
cryptonote_core does not depend on p2p anymore As a followon side effect, this makes a lot of inline code included only in particular cpp files (and instanciated when necessary.
2017-11-29 22:53:58 +00:00
#include "version.h"
moved all stuff to github
2014-03-03 22:07:58 +00:00
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
#undef MONERO_DEFAULT_LOG_CATEGORY
#define MONERO_DEFAULT_LOG_CATEGORY "cn"
moved all stuff to github
2014-03-03 22:07:58 +00:00
DISABLE_VS_WARNINGS(4355)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
#define MERROR_VER(x) MCERROR("verify", x)
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
#define BAD_SEMANTICS_TXES_MAX_SIZE 100
moved all stuff to github
2014-03-03 22:07:58 +00:00
namespace cryptonote
{
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
const command_line::arg_descriptor<bool, false> arg_testnet_on = {
"testnet"
, "Run on testnet. The wallet must be launched with --testnet flag."
, false
};
Stagenet
2018-02-16 11:04:04 +00:00
const command_line::arg_descriptor<bool, false> arg_stagenet_on = {
"stagenet"
, "Run on stagenet. The wallet must be launched with --stagenet flag."
, false
};
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
const command_line::arg_descriptor<bool> arg_regtest_on = {
"regtest"
, "Run in a regression testing mode."
, false
};
const command_line::arg_descriptor<difficulty_type> arg_fixed_difficulty = {
"fixed-difficulty"
, "Fixed difficulty used for testing."
, 0
};
Stagenet
2018-02-16 11:04:04 +00:00
const command_line::arg_descriptor<std::string, false, true, 2> arg_data_dir = {
options: remove testnet-* options
2018-01-21 15:29:55 +00:00
"data-dir"
, "Specify data directory"
, tools::get_default_data_dir()
Stagenet
2018-02-16 11:04:04 +00:00
, {{ &arg_testnet_on, &arg_stagenet_on }}
fix lambda compile error on openbsd
2018-03-27 13:47:57 +00:00
, [](std::array<bool, 2> testnet_stagenet, bool defaulted, std::string val)->std::string {
Stagenet
2018-02-16 11:04:04 +00:00
if (testnet_stagenet[0])
options: add testnet option dependencies
2018-01-22 01:49:51 +00:00
return (boost::filesystem::path(val) / "testnet").string();
Stagenet
2018-02-16 11:04:04 +00:00
else if (testnet_stagenet[1])
return (boost::filesystem::path(val) / "stagenet").string();
options: add testnet option dependencies
2018-01-22 01:49:51 +00:00
return val;
}
options: remove testnet-* options
2018-01-21 15:29:55 +00:00
};
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
const command_line::arg_descriptor<bool> arg_offline = {
"offline"
, "Do not listen for peers, nor connect to any"
};
cryptonote_core: add --disable-dns-checkpoints flag
2018-01-17 00:45:28 +00:00
const command_line::arg_descriptor<bool> arg_disable_dns_checkpoints = {
"disable-dns-checkpoints"
, "Do not retrieve checkpoints from DNS"
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
const command_line::arg_descriptor<size_t> arg_block_download_max_size = {
"block-download-max-size"
, "Set maximum size of block download queue in bytes (0 for default)"
, 0
};
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
static const command_line::arg_descriptor<bool> arg_test_drop_download = {
"test-drop-download"
, "For net tests: in download, discard ALL blocks instead checking/saving them (very fast)"
};
static const command_line::arg_descriptor<uint64_t> arg_test_drop_download_height = {
"test-drop-download-height"
Fix typos in various files
2018-03-01 11:36:19 +00:00
, "Like test-drop-download but discards only after around certain height"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
, 0
};
static const command_line::arg_descriptor<int> arg_test_dbg_lock_sleep = {
"test-dbg-lock-sleep"
, "Sleep time in ms, defaults to 0 (off), used to debug before/after locking mutex. Values 100 to 1000 are good for tests."
, 0
};
static const command_line::arg_descriptor<bool> arg_dns_checkpoints = {
"enforce-dns-checkpointing"
, "checkpoints from DNS server will be enforced"
, false
};
static const command_line::arg_descriptor<uint64_t> arg_fast_block_sync = {
"fast-block-sync"
, "Sync up most of the way by using embedded, known block hashes."
, 1
};
static const command_line::arg_descriptor<uint64_t> arg_prep_blocks_threads = {
"prep-blocks-threads"
, "Max number of threads to use when preparing block hashes in groups."
, 4
};
static const command_line::arg_descriptor<uint64_t> arg_show_time_stats = {
"show-time-stats"
, "Show time-stats when processing blocks/txs and disk synchronization."
, 0
};
static const command_line::arg_descriptor<size_t> arg_block_sync_size = {
"block-sync-size"
, "How many blocks to sync at once during chain synchronization (0 = adaptive)."
, 0
};
static const command_line::arg_descriptor<std::string> arg_check_updates = {
"check-updates"
, "Check for new versions of monero: [disabled|notify|download|update]"
, "notify"
};
static const command_line::arg_descriptor<bool> arg_fluffy_blocks = {
"fluffy-blocks"
core: add --no-fluffy-blocks, and enable fluffy blocks by default
2018-01-27 17:37:09 +00:00
, "Relay blocks as fluffy blocks (obsolete, now default)"
, true
};
static const command_line::arg_descriptor<bool> arg_no_fluffy_blocks = {
"no-fluffy-blocks"
, "Relay blocks as normal blocks"
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
, false
};
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 22:27:25 +00:00
static const command_line::arg_descriptor<bool> arg_pad_transactions = {
"pad-transactions"
, "Pad relayed transactions to help defend against traffic volume analysis"
, false
};
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
static const command_line::arg_descriptor<size_t> arg_max_txpool_weight = {
"max-txpool-weight"
, "Set maximum txpool weight in bytes."
, DEFAULT_TXPOOL_MAX_WEIGHT
tx_pool: add a max pool size, settable with --max-txpool-size
2018-01-29 21:20:24 +00:00
};
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 20:18:08 +00:00
static const command_line::arg_descriptor<std::string> arg_block_notify = {
"block-notify"
, "Run a program for each new block, '%s' will be replaced by the block hash"
, ""
};
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
static const command_line::arg_descriptor<bool> arg_prune_blockchain = {
"prune-blockchain"
, "Prune blockchain"
, false
};
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 14:29:28 +00:00
static const command_line::arg_descriptor<std::string> arg_reorg_notify = {
"reorg-notify"
, "Run a program for each reorg, '%s' will be replaced by the split height, "
"'%h' will be replaced by the new blockchain height, and '%n' will be "
"replaced by the number of new blocks in the new chain"
, ""
};
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 17:09:28 +00:00
static const command_line::arg_descriptor<std::string> arg_block_rate_notify = {
"block-rate-notify"
, "Run a program when the block rate undergoes large fluctuations. This might "
"be a sign of large amounts of hash rate going on and off the Monero network, "
"and thus be of potential interest in predicting attacks. %t will be replaced "
"by the number of minutes for the observation window, %b by the number of "
"blocks observed within that window, and %e by the number of blocks that was "
"expected in that window. It is suggested that this notification is used to "
"automatically increase the number of confirmations required before a payment "
"is acted upon."
, ""
};
moved all stuff to github
2014-03-03 22:07:58 +00:00
//-----------------------------------------------------------------------------------------------
core::core(i_cryptonote_protocol* pprotocol):
m_mempool(m_blockchain_storage),
m_blockchain_storage(m_mempool),
m_miner(this),
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
m_miner_address(boost::value_initialized<account_public_address>()),
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 20:50:13 +00:00
m_starter_message_showed(false),
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 05:01:58 +00:00
m_target_blockchain_height(0),
m_checkpoints_path(""),
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 20:30:47 +00:00
m_last_dns_checkpoints_update(0),
core: make update download cancellable
2017-02-26 21:00:38 +00:00
m_last_json_checkpoints_update(0),
core: new API to disable DNS checkpoint lookups
2017-07-26 09:18:44 +00:00
m_disable_dns_checkpoints(false),
core: fix use of uninitialised data
2018-03-18 23:36:20 +00:00
m_update_download(0),
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 14:31:34 +00:00
m_nettype(UNDEFINED),
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 22:27:25 +00:00
m_update_available(false),
m_pad_transactions(false)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
cryptonote_core: initialize checkpoint flag
2017-07-02 17:42:45 +00:00
m_checkpoints_updating.clear();
moved all stuff to github
2014-03-03 22:07:58 +00:00
set_cryptonote_protocol(pprotocol);
}
void core::set_cryptonote_protocol(i_cryptonote_protocol* pprotocol)
{
if(pprotocol)
m_pprotocol = pprotocol;
else
m_pprotocol = &m_protocol_stub;
}
//-----------------------------------------------------------------------------------
void core::set_checkpoints(checkpoints&& chk_pts)
{
m_blockchain_storage.set_checkpoints(std::move(chk_pts));
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 05:15:28 +00:00
}
//-----------------------------------------------------------------------------------
void core::set_checkpoints_file_path(const std::string& path)
{
m_checkpoints_path = path;
}
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 05:01:58 +00:00
//-----------------------------------------------------------------------------------
void core::set_enforce_dns_checkpoints(bool enforce_dns)
{
m_blockchain_storage.set_enforce_dns_checkpoints(enforce_dns);
}
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 05:15:28 +00:00
//-----------------------------------------------------------------------------------------------
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 05:01:58 +00:00
bool core::update_checkpoints()
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 05:15:28 +00:00
{
Stagenet
2018-02-16 11:04:04 +00:00
if (m_nettype != MAINNET || m_disable_dns_checkpoints) return true;
Disable DNS checkpoint updating on testnet
2015-04-08 22:07:46 +00:00
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 08:36:39 +00:00
if (m_checkpoints_updating.test_and_set()) return true;
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 05:01:58 +00:00
bool res = true;
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 20:30:47 +00:00
if (time(NULL) - m_last_dns_checkpoints_update >= 3600)
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 05:15:28 +00:00
{
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 20:30:47 +00:00
res = m_blockchain_storage.update_checkpoints(m_checkpoints_path, true);
m_last_dns_checkpoints_update = time(NULL);
m_last_json_checkpoints_update = time(NULL);
}
else if (time(NULL) - m_last_json_checkpoints_update >= 600)
{
res = m_blockchain_storage.update_checkpoints(m_checkpoints_path, false);
m_last_json_checkpoints_update = time(NULL);
reload checkpoints file every ~hr and print if any fail also some other minor bug squashing and code formatting
2014-09-25 05:15:28 +00:00
}
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 19:24:43 +00:00
DNS Checkpoint updating-related fixes/changes Only one thread will be doing the updating. Two valid responses must match, and the first two that match will be used.
2015-04-22 08:36:39 +00:00
m_checkpoints_updating.clear();
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 19:24:43 +00:00
// if anything fishy happened getting new checkpoints, bring down the house
if (!res)
{
graceful_exit();
}
DNS checkpoint updating added, and daemon flag to enforce them The daemon should now check for updated checkpoints from checkpoints.moneropulse.org as well as from the configured json file every ~1hr (and on launch). The daemon now has a flag to enable enforcing these checkpoints (rather than just printing a warning when they fail). TODO: an easily configurable list of DNS servers to check for checkpoints as opposed to the hard-coded "checkpoints.moneropulse.org"
2014-09-26 05:01:58 +00:00
return res;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 09:11:20 +00:00
void core::stop()
{
daemon: stop miner before we bring the whole thing down This avoids the miner erroring out trying to submit blocks to a core that's already shut down (and avoids pegging the CPU while we're busy shutting down).
2018-11-30 14:53:38 +00:00
m_miner.stop();
Fix delayed exit when syncing
2016-12-04 12:27:45 +00:00
m_blockchain_storage.cancel();
core: make update download cancellable
2017-02-26 21:00:38 +00:00
tools::download_async_handle handle;
{
boost::lock_guard<boost::mutex> lock(m_update_mutex);
handle = m_update_download;
m_update_download = 0;
}
if (handle)
tools::download_cancel(handle);
RPC calls for background daemon added in The RPC calls the daemon executable uses to talk to the running daemon instance have mostly been added back in. Rate limiting has not been added in upstream, but is on its way in a separate effort, so those calls are still NOPed out.
2015-02-05 09:11:20 +00:00
}
//-----------------------------------------------------------------------------------
Register daemon command line arguments to core if they're used in core This fixes coretests, which does not register daemon specific arguments, but uses core, which uses those arguments. Also gets rid of an unwanted dependency on daemon code from core.
2015-12-08 23:06:29 +00:00
void core::init_options(boost::program_options::options_description& desc)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
options: remove testnet-* options
2018-01-21 15:29:55 +00:00
command_line::add_arg(desc, arg_data_dir);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
command_line::add_arg(desc, arg_test_drop_download);
command_line::add_arg(desc, arg_test_drop_download_height);
command_line::add_arg(desc, arg_testnet_on);
Stagenet
2018-02-16 11:04:04 +00:00
command_line::add_arg(desc, arg_stagenet_on);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
command_line::add_arg(desc, arg_regtest_on);
command_line::add_arg(desc, arg_fixed_difficulty);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
command_line::add_arg(desc, arg_dns_checkpoints);
command_line::add_arg(desc, arg_prep_blocks_threads);
command_line::add_arg(desc, arg_fast_block_sync);
command_line::add_arg(desc, arg_show_time_stats);
command_line::add_arg(desc, arg_block_sync_size);
command_line::add_arg(desc, arg_check_updates);
command_line::add_arg(desc, arg_fluffy_blocks);
core: add --no-fluffy-blocks, and enable fluffy blocks by default
2018-01-27 17:37:09 +00:00
command_line::add_arg(desc, arg_no_fluffy_blocks);
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
command_line::add_arg(desc, arg_test_dbg_lock_sleep);
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
command_line::add_arg(desc, arg_offline);
cryptonote_core: add --disable-dns-checkpoints flag
2018-01-17 00:45:28 +00:00
command_line::add_arg(desc, arg_disable_dns_checkpoints);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
command_line::add_arg(desc, arg_block_download_max_size);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
command_line::add_arg(desc, arg_max_txpool_weight);
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 22:27:25 +00:00
command_line::add_arg(desc, arg_pad_transactions);
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 20:18:08 +00:00
command_line::add_arg(desc, arg_block_notify);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
command_line::add_arg(desc, arg_prune_blockchain);
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 14:29:28 +00:00
command_line::add_arg(desc, arg_reorg_notify);
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 17:09:28 +00:00
command_line::add_arg(desc, arg_block_rate_notify);
core: add p2p bind port options from net_node They're now used by core to determine the data directory to use for the txpool directory. This fixes an assert in the core tests, which don't use the RPC server, which normally initializes the P2P code.
2017-02-09 21:29:35 +00:00
Fix core_tests breaking on startup You're wondering how this fixes core tests, aren't you... It prevents the miner (initialized by cryptonote::core) from breaking trying to access arguments that were not added. Since the tests don't use the miner directly, it makes more sense to have cryptonote_core add those, since it also uses the miner.
2017-02-21 15:06:07 +00:00
miner::init_options(desc);
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 10:14:42 +00:00
BlockchainDB::init_options(desc);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
bool core::handle_command_line(const boost::program_options::variables_map& vm)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Stagenet
2018-02-16 11:04:04 +00:00
if (m_nettype != FAKECHAIN)
{
const bool testnet = command_line::get_arg(vm, arg_testnet_on);
const bool stagenet = command_line::get_arg(vm, arg_stagenet_on);
m_nettype = testnet ? TESTNET : stagenet ? STAGENET : MAINNET;
}
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
options: remove testnet-* options
2018-01-21 15:29:55 +00:00
m_config_folder = command_line::get_arg(vm, arg_data_dir);
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
auto data_dir = boost::filesystem::path(m_config_folder);
Stagenet
2018-02-16 11:04:04 +00:00
if (m_nettype == MAINNET)
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
{
cryptonote::checkpoints checkpoints;
Stagenet
2018-02-16 11:04:04 +00:00
if (!checkpoints.init_default_checkpoints(m_nettype))
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
{
Revert "Print stack trace upon exceptions" Ain't nobody got time for link/cmake skullduggery. This reverts commit fff238ec94ac6d45fc18c315d7bc590ddfaad63d.
2016-03-21 10:12:12 +00:00
throw std::runtime_error("Failed to initialize checkpoints");
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
}
set_checkpoints(std::move(checkpoints));
boost::filesystem::path json(JSON_HASH_FILE_NAME);
boost::filesystem::path checkpoint_json_hashfile_fullpath = data_dir / json;
set_checkpoints_file_path(checkpoint_json_hashfile_fullpath.string());
}
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
set_enforce_dns_checkpoints(command_line::get_arg(vm, arg_dns_checkpoints));
test_drop_download_height(command_line::get_arg(vm, arg_test_drop_download_height));
core: add --no-fluffy-blocks, and enable fluffy blocks by default
2018-01-27 17:37:09 +00:00
m_fluffy_blocks_enabled = !get_arg(vm, arg_no_fluffy_blocks);
protocol: option to pad transaction relay to the next kB To help protect one's privacy from traffic volume analysis for people using Tor or I2P. This will really fly once we relay txes on a timer rather than on demand, though. Off by default for now since it's wasteful and doesn't bring anything until I2P's in.
2018-11-02 22:27:25 +00:00
m_pad_transactions = get_arg(vm, arg_pad_transactions);
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
m_offline = get_arg(vm, arg_offline);
cryptonote_core: add --disable-dns-checkpoints flag
2018-01-17 00:45:28 +00:00
m_disable_dns_checkpoints = get_arg(vm, arg_disable_dns_checkpoints);
core: add --no-fluffy-blocks, and enable fluffy blocks by default
2018-01-27 17:37:09 +00:00
if (!command_line::is_arg_defaulted(vm, arg_fluffy_blocks))
MWARNING(arg_fluffy_blocks.name << " is obsolete, it is now default");
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
if (command_line::get_arg(vm, arg_test_drop_download) == true)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
test_drop_download();
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
epee::debug::g_test_dbg_lock_sleep() = command_line::get_arg(vm, arg_test_dbg_lock_sleep);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
uint64_t core::get_current_blockchain_height() const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_current_blockchain_height();
}
//-----------------------------------------------------------------------------------------------
get_blockchain_top now returns void It was always returning true, and could not be foreseen to usefully return errors in the future. This silences CID 162652 as well as saves some checking code in a few places.
2017-09-09 11:06:24 +00:00
void core::get_blockchain_top(uint64_t& height, crypto::hash& top_id) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
top_id = m_blockchain_storage.get_tail_id(height);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks, std::vector<cryptonote::blobdata>& txs) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks, txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<std::pair<cryptonote::blobdata,block>>& blocks) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_blocks(start_offset, count, blocks);
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_blocks(uint64_t start_offset, size_t count, std::vector<block>& blocks) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<std::pair<cryptonote::blobdata, cryptonote::block>> bs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
if (!m_blockchain_storage.get_blocks(start_offset, count, bs))
return false;
for (const auto &b: bs)
blocks.push_back(b.second);
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<cryptonote::blobdata>& txs, std::vector<crypto::hash>& missed_txs) const
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
{
return m_blockchain_storage.get_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool core::get_split_transactions_blobs(const std::vector<crypto::hash>& txs_ids, std::vector<std::tuple<crypto::hash, cryptonote::blobdata, crypto::hash, cryptonote::blobdata>>& txs, std::vector<crypto::hash>& missed_txs) const
{
return m_blockchain_storage.get_split_transactions_blobs(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
rpc: add a new RPC to get current txpool backlog (sizes and fees)
2017-08-26 15:23:31 +00:00
bool core::get_txpool_backlog(std::vector<tx_backlog_entry>& backlog) const
{
m_mempool.get_transaction_backlog(backlog);
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_transactions(const std::vector<crypto::hash>& txs_ids, std::vector<transaction>& txs, std::vector<crypto::hash>& missed_txs) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_transactions(txs_ids, txs, missed_txs);
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_alternative_blocks(std::vector<block>& blocks) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_alternative_blocks(blocks);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
size_t core::get_alternative_blocks_count() const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_alternative_blocks_count();
}
//-----------------------------------------------------------------------------------------------
remove some unused code Found by codacy.com
2018-11-23 13:47:51 +00:00
bool core::init(const boost::program_options::variables_map& vm, const cryptonote::test_options *test_options, const GetCheckpointsCallback& get_checkpoints/* = nullptr */)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Add start_time to get_info methods and show uptime
2017-01-08 23:50:29 +00:00
start_time = std::time(nullptr);
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
const bool regtest = command_line::get_arg(vm, arg_regtest_on);
if (test_options != NULL || regtest)
Stagenet
2018-02-16 11:04:04 +00:00
{
m_nettype = FAKECHAIN;
}
Daemonize changes pulled in -- daemon builds many RPC functions added by the daemonize changes (and related changes on the upstream dev branch that were not merged) were commented out (apart from return). Other than that, this *should* work...at any rate, it builds, and that's something.
2015-01-29 22:10:53 +00:00
bool r = handle_command_line(vm);
Separate data per P2P port poolstate.bin and p2pstate.bin are stored in .bitmonero/ if the default P2P port is being used. If another port is used both files are stored in .bitmonero/PORTNUMBER/.
2017-01-28 00:36:39 +00:00
move db specific options to BlockchainDB Avoids common depending on blockchain_db, which can cause link errors.
2017-08-29 10:14:42 +00:00
std::string db_type = command_line::get_arg(vm, cryptonote::arg_db_type);
std::string db_sync_mode = command_line::get_arg(vm, cryptonote::arg_db_sync_mode);
bool db_salvage = command_line::get_arg(vm, cryptonote::arg_db_salvage) != 0;
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
bool fast_sync = command_line::get_arg(vm, arg_fast_block_sync) != 0;
uint64_t blocks_threads = command_line::get_arg(vm, arg_prep_blocks_threads);
std::string check_updates_string = command_line::get_arg(vm, arg_check_updates);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
size_t max_txpool_weight = command_line::get_arg(vm, arg_max_txpool_weight);
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool prune_blockchain = command_line::get_arg(vm, arg_prune_blockchain);
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 17:09:00 +00:00
boost::filesystem::path folder(m_config_folder);
Stagenet
2018-02-16 11:04:04 +00:00
if (m_nettype == FAKECHAIN)
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 17:09:00 +00:00
folder /= "fake";
core: move the db lock to the data directory Locking just one db turns out to not have been a good idea, since the pool and p2p state fdles have to be used anyway. Also ensure the directory exists before tring to lock.
2016-02-04 18:09:45 +00:00
// make sure the data directory exists, and try to lock it
CHECK_AND_ASSERT_MES (boost::filesystem::exists(folder) || boost::filesystem::create_directories(folder), false,
std::string("Failed to create directory ").append(folder.string()).c_str());
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 17:09:00 +00:00
// check for blockchain.bin
try
{
const boost::filesystem::path old_files = folder;
if (boost::filesystem::exists(old_files / "blockchain.bin"))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MWARNING("Found old-style blockchain.bin in " << old_files.string());
MWARNING("Monero now uses a new format. You can either remove blockchain.bin to start syncing");
MWARNING("the blockchain anew, or use monero-blockchain-export and monero-blockchain-import to");
MWARNING("convert your existing blockchain.bin to the new format. See README.md for instructions.");
Nicer looking exit when blockchain.bin is found Do not print the exception message, and write the important bit in red, since people will only read the last line otherwise.
2015-12-31 17:09:00 +00:00
return false;
}
}
// folder might not be a directory, etc, etc
catch (...) { }
cryptonote_core: fix db leak on error
2017-12-13 14:04:19 +00:00
std::unique_ptr<BlockchainDB> db(new_db(db_type));
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
if (db == NULL)
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
{
Fix a few minor typos
2016-12-04 13:13:54 +00:00
LOG_ERROR("Attempted to use non-existent database type");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
return false;
}
folder /= db->get_db_name();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MGINFO("Loading blockchain from folder " << folder.string() << " ...");
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
const std::string filename = folder.string();
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 10:09:35 +00:00
// default to fast:async:1 if overridden
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 18:36:51 +00:00
blockchain_db_sync_mode sync_mode = db_defaultsync;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 10:09:35 +00:00
bool sync_on_blocks = true;
uint64_t sync_threshold = 1;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
if (m_nettype == FAKECHAIN)
{
// reset the db by removing the database file before opening it
if (!db->remove_data_file(filename))
{
MERROR("Failed to remove data file in " << filename);
return false;
}
}
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
try
{
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
uint64_t db_flags = 0;
std::vector<std::string> options;
boost::trim(db_sync_mode);
boost::split(options, db_sync_mode, boost::is_any_of(" :"));
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
const bool db_sync_mode_is_default = command_line::is_arg_defaulted(vm, cryptonote::arg_db_sync_mode);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
for(const auto &option : options)
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MDEBUG("option: " << option);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 03:38:55 +00:00
// default to fast:async:1
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
uint64_t DEFAULT_FLAGS = DBF_FAST;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
if(options.size() == 0)
{
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 03:38:55 +00:00
// default to fast:async:1
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
db_flags = DEFAULT_FLAGS;
}
bool safemode = false;
if(options.size() >= 1)
{
if(options[0] == "safe")
{
safemode = true;
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
db_flags = DBF_SAFE;
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_nosync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
}
else if(options[0] == "fast")
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 18:36:51 +00:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
db_flags = DBF_FAST;
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Toggle SAFE syncmode on and off automatically If monerod is started with default sync mode, set it to SAFE after synchronization completes. Set it back to FAST if synchronization restarts (e.g. because another peer has a longer blockchain). If monerod is started with an explicit sync mode, none of this automation takes effect.
2017-08-19 18:36:51 +00:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
else if(options[0] == "fastest")
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 03:38:55 +00:00
{
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
db_flags = DBF_FASTEST;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 10:09:35 +00:00
sync_threshold = 1000; // default to fastest:async:1000
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
Tweak default db-sync-mode to fast:async:1 fsync the DB asynchronously, to allow block download/verification to proceed while syncing. Sync after every batch. Note that "fastest" still defaults to fastest:async:1000.
2017-01-01 03:38:55 +00:00
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
else
db_flags = DEFAULT_FLAGS;
}
if(options.size() >= 2 && !safemode)
{
if(options[1] == "sync")
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_sync;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
else if(options[1] == "async")
core: fix automatic safe db sync mode switching
2018-05-28 11:12:19 +00:00
sync_mode = db_sync_mode_is_default ? db_defaultsync : db_async;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
}
if(options.size() >= 3 && !safemode)
{
Remove blocks_per_sync limits The code used to cap at 5000 blocks per sync. It also treated 0 as 1. Remove these checks; if specified as 0 do no periodic syncs at all. Then the user is responsible for syncing in some external process.
2016-08-29 15:42:52 +00:00
char *endptr;
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 10:09:35 +00:00
uint64_t threshold = strtoull(options[2].c_str(), &endptr, 0);
if (*endptr == '\0' || !strcmp(endptr, "blocks"))
{
sync_on_blocks = true;
sync_threshold = threshold;
}
else if (!strcmp(endptr, "bytes"))
{
sync_on_blocks = false;
sync_threshold = threshold;
}
else
{
LOG_ERROR("Invalid db sync mode: " << options[2]);
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
}
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-30 22:59:52 +00:00
if (db_salvage)
DB cleanup Hide LMDB-specific stuff behind blockchain_db.h. Nobody besides blockchain_db.cpp should ever be including DB-specific headers any more.
2017-08-19 14:27:13 +00:00
db_flags |= DBF_SALVAGE;
core: add --db-salvage command line flag Use to load the database when the primary meta page is corrupted
2017-07-30 22:59:52 +00:00
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
db->open(filename, db_flags);
if(!db->m_open)
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return false;
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
}
catch (const DB_ERROR& e)
{
Moved logging to target functions rather than caller
2016-10-03 21:11:00 +00:00
LOG_ERROR("Error opening database: " << e.what());
Steps toward multiple dbs available -- working There will need to be some more refactoring for these changes to be considered complete/correct, but for now it's working. new daemon cli argument "--db-type", works for LMDB and BerkeleyDB. A good deal of refactoring is also present in this commit, namely Blockchain no longer instantiates BlockchainDB, but rather is passed a pointer to an already-instantiated BlockchainDB on init().
2015-03-25 15:41:30 +00:00
return false;
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
m_blockchain_storage.set_user_options(blocks_threads,
core: sync database based on bytes added, not blocks added Blocks have a very wide range, whereas actual size is the relevant quantity to consider when syncing
2018-08-01 10:09:35 +00:00
sync_on_blocks, sync_threshold, sync_mode, fast_sync);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
Merge pull request #4333 73403004 add --block-notify to monerod and --tx-notify to monero-wallet-{cli,rpc} (moneromooo-monero)
2018-09-29 20:18:08 +00:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_notify))
m_blockchain_storage.set_block_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_block_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block notify spec");
}
blockchain: add --reorg-notify This will trigger if a reorg is seen. This may be used to do things like stop automated withdrawals on large reorgs. %s is replaced by the height at the split point %h is replaced by the height of the new chain %n is replaced by the number of new blocks after the reorg
2019-01-09 14:29:28 +00:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_reorg_notify))
m_blockchain_storage.set_reorg_notify(std::shared_ptr<tools::Notify>(new tools::Notify(command_line::get_arg(vm, arg_reorg_notify).c_str())));
}
catch (const std::exception &e)
{
MERROR("Failed to parse reorg notify spec");
}
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 17:09:28 +00:00
try
{
if (!command_line::is_arg_defaulted(vm, arg_block_rate_notify))
m_block_rate_notify.reset(new tools::Notify(command_line::get_arg(vm, arg_block_rate_notify).c_str()));
}
catch (const std::exception &e)
{
MERROR("Failed to parse block rate notify spec: " << e.what());
}
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
const std::pair<uint8_t, uint64_t> regtest_hard_forks[3] = {std::make_pair(1, 0), std::make_pair(Blockchain::get_hard_fork_heights(MAINNET).back().version, 1), std::make_pair(0, 0)};
const cryptonote::test_options regtest_test_options = {
ArticMine's new block weight algorithm This curbs runaway growth while still allowing substantial spikes in block weight Original specification from ArticMine: here is the scaling proposal Define: LongTermBlockWeight Before fork: LongTermBlockWeight = BlockWeight At or after fork: LongTermBlockWeight = min(BlockWeight, 1.4*LongTermEffectiveMedianBlockWeight) Note: To avoid possible consensus issues over rounding the LongTermBlockWeight for a given block should be calculated to the nearest byte, and stored as a integer in the block itself. The stored LongTermBlockWeight is then used for future calculations of the LongTermEffectiveMedianBlockWeight and not recalculated each time. Define: LongTermEffectiveMedianBlockWeight LongTermEffectiveMedianBlockWeight = max(300000, MedianOverPrevious100000Blocks(LongTermBlockWeight)) Change Definition of EffectiveMedianBlockWeight From (current definition) EffectiveMedianBlockWeight = max(300000, MedianOverPrevious100Blocks(BlockWeight)) To (proposed definition) EffectiveMedianBlockWeight = min(max(300000, MedianOverPrevious100Blocks(BlockWeight)), 50*LongTermEffectiveMedianBlockWeight) Notes: 1) There are no other changes to the existing penalty formula, median calculation, fees etc. 2) There is the requirement to store the LongTermBlockWeight of a block unencrypted in the block itself. This is to avoid possible consensus issues over rounding and also to prevent the calculations from becoming unwieldy as we move away from the fork. 3) When the EffectiveMedianBlockWeight cap is reached it is still possible to mine blocks up to 2x the EffectiveMedianBlockWeight by paying the corresponding penalty. Note: the long term block weight is stored in the database, but not in the actual block itself, since it requires recalculating anyway for verification.
2019-01-21 17:18:50 +00:00
regtest_hard_forks,
0
add --regtest and --fixed-difficulty for regression testing on_generateblocks RPC call combines functionality from the on_getblocktemplate and on_submitblock RPC calls to allow rapid block creation. Difficulty is set permanently to 1 for regtest. Makes use of FAKECHAIN network type, but takes hard fork heights from mainchain Default reserve_size in generate_blocks RPC call is now 1. If it is 0, the following error occurs 'Failed to calculate offset for'. Queries hard fork heights info of other network types
2018-06-14 19:11:49 +00:00
};
const difficulty_type fixed_difficulty = command_line::get_arg(vm, arg_fixed_difficulty);
blocks: use auto-generated .c files instead of 'LD -r -b binary'
2018-10-09 12:33:39 +00:00
r = m_blockchain_storage.init(db.release(), m_nettype, m_offline, regtest ? &regtest_test_options : test_options, fixed_difficulty, get_checkpoints);
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
r = m_mempool.init(max_txpool_weight);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize memory pool");
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 15:09:17 +00:00
// now that we have a valid m_blockchain_storage, we can clean out any
// transactions in the pool that do not conform to the current fork
m_mempool.validate(m_blockchain_storage.get_current_hard_fork_version());
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
bool show_time_stats = command_line::get_arg(vm, arg_show_time_stats) != 0;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
m_blockchain_storage.set_show_time_stats(show_time_stats);
moved all stuff to github
2014-03-03 22:07:58 +00:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize blockchain storage");
move cryptonote command line options to cryptonote_core Those have no reason to be in a generic module
2017-10-28 15:06:43 +00:00
block_sync_size = command_line::get_arg(vm, arg_block_sync_size);
core: make the sync chunk block count overridable
2016-09-24 16:00:19 +00:00
core: add a message when loading checkpoints at init time This uses DNS, which can take a while, so it's useful to know this is the culprit when loading pauses
2017-07-31 05:43:13 +00:00
MGINFO("Loading checkpoints");
Fixed segfault with checkpoints loading Bounds checking on blockchain_storage' m_blocks.size() when validating against checkpoints. Also moved initial json & DNS checkpoints load to after blockchain init.
2014-09-27 21:46:12 +00:00
// load json & DNS checkpoints, and verify them
// with respect to what blocks we already have
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 20:30:47 +00:00
CHECK_AND_ASSERT_MES(update_checkpoints(), false, "One or more checkpoints loaded from json or dns conflicted with existing checkpoints.");
Fixed segfault with checkpoints loading Bounds checking on blockchain_storage' m_blocks.size() when validating against checkpoints. Also moved initial json & DNS checkpoints load to after blockchain init.
2014-09-27 21:46:12 +00:00
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
// DNS versions checking
if (check_updates_string == "disabled")
check_updates_level = UPDATES_DISABLED;
else if (check_updates_string == "notify")
check_updates_level = UPDATES_NOTIFY;
else if (check_updates_string == "download")
check_updates_level = UPDATES_DOWNLOAD;
else if (check_updates_string == "update")
check_updates_level = UPDATES_UPDATE;
else {
MERROR("Invalid argument to --dns-versions-check: " << check_updates_string);
return false;
}
Stagenet
2018-02-16 11:04:04 +00:00
r = m_miner.init(vm, m_nettype);
Merge pull request #749 bfd4a28 Update BlockchainDB documentation (Thomas Winget) 797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget) c835215 remove defunct code from cryptonote::core (Thomas Winget) 50dba6d cryptonote::core doxygen documentation (Thomas Winget) 8ac329d doxygen documentation for difficulty functions (Thomas Winget) 540a76c Move checkpoint functions into checkpoints class (Thomas Winget) 1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget) 89c24ac Remove unnecessary or defunct code (Thomas Winget) ab0ed14 doxygen include private and static members (Thomas Winget) 3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 06:22:06 +00:00
CHECK_AND_ASSERT_MES(r, false, "Failed to initialize miner instance");
moved all stuff to github
2014-03-03 22:07:58 +00:00
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
if (prune_blockchain)
{
// display a message if the blockchain is not pruned yet
if (m_blockchain_storage.get_current_blockchain_height() > 1 && !m_blockchain_storage.get_blockchain_pruning_seed())
MGINFO("Pruning blockchain...");
CHECK_AND_ASSERT_MES(m_blockchain_storage.prune_blockchain(), false, "Failed to prune blockchain");
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
return load_state_data();
}
//-----------------------------------------------------------------------------------------------
bool core::set_genesis_block(const block& b)
{
return m_blockchain_storage.reset_and_set_genesis_block(b);
}
//-----------------------------------------------------------------------------------------------
bool core::load_state_data()
{
// may be some code later
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::deinit()
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
m_miner.stop();
m_mempool.deinit();
Removed all code related to fast_exit
2016-10-03 01:06:55 +00:00
m_blockchain_storage.deinit();
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
void core::test_drop_download()
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
m_test_drop_download = false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
}
//-----------------------------------------------------------------------------------------------
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
void core::test_drop_download_height(uint64_t height)
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
m_test_drop_download_height = height;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::get_test_drop_download() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return m_test_drop_download;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::get_test_drop_download_height() const
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
{
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
if (m_test_drop_download_height == 0)
return true;
if (get_blockchain_storage().get_current_blockchain_height() <= m_test_drop_download_height)
return true;
2014 network limit 1.2 +utils +toc -doc -drmonero new update of the pr with network limits more debug options: discarding downloaded blocks all or after given height. trying to trigger the locking errors. debug levels polished/tuned to sane values. debug/logging improved. warning: this pr should be correct code, but it could make an existing (in master version) locking error appear more often. it's a race on the list (map) of peers, e.g. between closing/deleting them versus working on them in net-limit sleep in sending chunk. the bug is not in this code/this pr, but in the master version. the locking problem of master will be fixed in other pr. problem is ub, and in practice is seems to usually cause program abort (tested on debian stable with updated gcc). see --help for option to add sleep to trigger the error faster.
2015-02-20 21:28:03 +00:00
Replace tabs with two spaces for consistency with rest of codebase Remove trailing whitespace in same files.
2015-12-14 04:54:39 +00:00
return false;
2014 network limit 1.1 +utils +toc -doc -drmonero Update of the PR with network limits works very well for all speeds (but remember that low download speed can stop upload because we then slow down downloading of blockchain requests too) more debug options fixed pedantic warnings in our code should work again on Mac OS X and FreeBSD fixed warning about size_t tested on Debian, Ubuntu, Windows(testing now) TCP options and ToS (QoS) flag FIXED peer number limit FIXED some spikes in ingress/download FIXED problems when other up and down limit
2015-02-12 19:59:39 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
//-----------------------------------------------------------------------------------------------
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
bool core::handle_incoming_tx_pre(const blobdata& tx_blob, tx_verification_context& tvc, cryptonote::transaction &tx, crypto::hash &tx_hash, crypto::hash &tx_prefixt_hash, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
tvc = boost::value_initialized<tx_verification_context>();
if(tx_blob.size() > get_max_tx_size())
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, too big size " << tx_blob.size() << ", rejected");
moved all stuff to github
2014-03-03 22:07:58 +00:00
tvc.m_verifivation_failed = true;
Convey tx verification failure reasons to the RPC client This allows appropriate action to be taken, like displaying the reason to the user. Do just that in simplewallet, which should help a lot in determining why users fail to send. Also make it so a tx which is accepted but not relayed is seen as a success rather than a failure.
2016-03-27 11:35:36 +00:00
tvc.m_too_big = true;
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
remove "using namespace std" from headers It's nasty, and actually breaks on Solaris, where if.h fails to build due to: struct map *if_memmap;
2017-10-10 14:47:08 +00:00
tx_hash = crypto::null_hash;
tx_prefixt_hash = crypto::null_hash;
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(!parse_tx_from_blob(tx, tx_hash, tx_prefixt_hash, tx_blob))
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to parse, rejected");
moved all stuff to github
2014-03-03 22:07:58 +00:00
tvc.m_verifivation_failed = true;
return false;
}
//std::cout << "!"<< tx.vin.size() << std::endl;
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
bad_semantics_txes_lock.lock();
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
for (int idx = 0; idx < 2; ++idx)
core: cache tx hashes of failing semantics txes
2017-01-22 16:13:31 +00:00
{
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
if (bad_semantics_txes[idx].find(tx_hash) != bad_semantics_txes[idx].end())
{
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
bad_semantics_txes_lock.unlock();
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
LOG_PRINT_L1("Transaction already seen with bad semantics, rejected");
tvc.m_verifivation_failed = true;
return false;
}
core: cache tx hashes of failing semantics txes
2017-01-22 16:13:31 +00:00
}
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
bad_semantics_txes_lock.unlock();
core: cache tx hashes of failing semantics txes
2017-01-22 16:13:31 +00:00
Condition v2 txes on v3 hard fork
2016-06-29 22:00:20 +00:00
uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
const size_t max_tx_version = version == 1 ? 1 : 2;
if (tx.version == 0 || tx.version > max_tx_version)
{
// v2 is the latest one we know
tvc.m_verifivation_failed = true;
return false;
}
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_tx_post(const blobdata& tx_blob, tx_verification_context& tvc, cryptonote::transaction &tx, crypto::hash &tx_hash, crypto::hash &tx_prefixt_hash, bool keeped_by_block, bool relayed, bool do_not_relay)
{
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(!check_tx_syntax(tx))
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to check tx " << tx_hash << " syntax, rejected");
moved all stuff to github
2014-03-03 22:07:58 +00:00
tvc.m_verifivation_failed = true;
return false;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
void core::set_semantics_failed(const crypto::hash &tx_hash)
{
LOG_PRINT_L1("WRONG TRANSACTION BLOB, Failed to check tx " << tx_hash << " semantic, rejected");
bad_semantics_txes_lock.lock();
bad_semantics_txes[0].insert(tx_hash);
if (bad_semantics_txes[0].size() >= BAD_SEMANTICS_TXES_MAX_SIZE)
{
std::swap(bad_semantics_txes[0], bad_semantics_txes[1]);
bad_semantics_txes[0].clear();
}
bad_semantics_txes_lock.unlock();
}
//-----------------------------------------------------------------------------------------------
require canonical multi output bulletproof layout
2018-03-31 09:49:53 +00:00
static bool is_canonical_bulletproof_layout(const std::vector<rct::Bulletproof> &proofs)
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
if (proofs.size() != 1)
return false;
const size_t sz = proofs[0].V.size();
if (sz == 0 || sz > BULLETPROOF_MAX_OUTPUTS)
return false;
require canonical multi output bulletproof layout
2018-03-31 09:49:53 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
bool core::handle_incoming_tx_accumulated_batch(std::vector<tx_verification_batch_info> &tx_info, bool keeped_by_block)
{
bool ret = true;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 08:45:55 +00:00
if (keeped_by_block && get_blockchain_storage().is_within_compiled_block_hash_area())
{
MTRACE("Skipping semantics check for tx kept by block in embedded hash area");
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
return true;
blockchain: skip checking tx semantics in embedded block hash range If the txes are bad, this'll be picked up by the block hash mismatch since the tx merkle root is part of the block hash.
2017-07-30 08:45:55 +00:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
std::vector<const rct::rctSig*> rvv;
for (size_t n = 0; n < tx_info.size(); ++n)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
if (!check_tx_semantic(*tx_info[n].tx, keeped_by_block))
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
{
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
continue;
}
if (tx_info[n].tx->version < 2)
continue;
const rct::rctSig &rv = tx_info[n].tx->rct_signatures;
switch (rv.type) {
case rct::RCTTypeNull:
// coinbase should not come here, so we reject for all other types
MERROR_VER("Unexpected Null rctSig type");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
case rct::RCTTypeSimple:
if (!rct::verRctSemanticsSimple(rv))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeFull:
if (!rct::verRct(rv, true))
{
MERROR_VER("rct signature semantics check failed");
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
break;
case rct::RCTTypeBulletproof:
add a bulletproof version, new bulletproof type, and rct config This makes it easier to modify the bulletproof format
2019-01-06 13:47:16 +00:00
case rct::RCTTypeBulletproof2:
require canonical multi output bulletproof layout
2018-03-31 09:49:53 +00:00
if (!is_canonical_bulletproof_layout(rv.p.bulletproofs))
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
MERROR_VER("Bulletproof does not have canonical form");
require canonical multi output bulletproof layout
2018-03-31 09:49:53 +00:00
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
rvv.push_back(&rv); // delayed batch verification
break;
default:
MERROR_VER("Unknown rct type: " << rv.type);
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
break;
}
}
if (!rvv.empty() && !rct::verRctSemanticsSimple(rvv))
{
LOG_PRINT_L1("One transaction among this group has bad semantics, verifying one at a time");
ret = false;
const bool assumed_bad = rvv.size() == 1; // if there's only one tx, it must be the bad one
for (size_t n = 0; n < tx_info.size(); ++n)
{
if (!tx_info[n].result)
continue;
add a bulletproof version, new bulletproof type, and rct config This makes it easier to modify the bulletproof format
2019-01-06 13:47:16 +00:00
if (tx_info[n].tx->rct_signatures.type != rct::RCTTypeBulletproof && tx_info[n].tx->rct_signatures.type != rct::RCTTypeBulletproof2)
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
continue;
if (assumed_bad || !rct::verRctSemanticsSimple(tx_info[n].tx->rct_signatures))
{
set_semantics_failed(tx_info[n].tx_hash);
tx_info[n].tvc.m_verifivation_failed = true;
tx_info[n].result = false;
}
core: bound the amount of entries in bad tx semantics cache This is to prevent unbounded memory use. Since I don't think there is a container that has quick insert, quick lookup, and automatic FIFO, I use two and swap every N, clearing the oldest one.
2017-02-23 20:15:52 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
return ret;
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::handle_incoming_txs(const std::vector<blobdata>& tx_blobs, std::vector<tx_verification_context>& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
{
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 12:01:45 +00:00
TRY_ENTRY();
core: lock incoming tx lock when checking the txpool and chain This gets rid of an innocuous race trying to add the same tx twice to the txpool
2018-05-18 08:38:38 +00:00
CRITICAL_REGION_LOCAL(m_incoming_tx_lock);
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 12:01:45 +00:00
remove some unused code Found by codacy.com
2018-11-23 13:47:51 +00:00
struct result { bool res; cryptonote::transaction tx; crypto::hash hash; crypto::hash prefix_hash; };
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
std::vector<result> results(tx_blobs.size());
tvc.resize(tx_blobs.size());
core: remove threadpool dependency from header
2018-04-23 17:55:58 +00:00
tools::threadpool& tpool = tools::threadpool::getInstance();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
tools::threadpool::waiter waiter;
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<blobdata>::const_iterator it = tx_blobs.begin();
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
core: remove threadpool dependency from header
2018-04-23 17:55:58 +00:00
tpool.submit(&waiter, [&, i, it] {
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
try
{
results[i].res = handle_incoming_tx_pre(*it, tvc[i], results[i].tx, results[i].hash, results[i].prefix_hash, keeped_by_block, relayed, do_not_relay);
}
catch (const std::exception &e)
{
MERROR_VER("Exception in handle_incoming_tx_pre: " << e.what());
results[i].res = false;
}
});
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 10:44:47 +00:00
waiter.wait(&tpool);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
it = tx_blobs.begin();
core: don't verify range proofs multiple times
2018-10-17 16:22:27 +00:00
std::vector<bool> already_have(tx_blobs.size(), false);
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
if (!results[i].res)
continue;
if(m_mempool.have_tx(results[i].hash))
{
LOG_PRINT_L2("tx " << results[i].hash << "already have transaction in tx_pool");
core: don't verify range proofs multiple times
2018-10-17 16:22:27 +00:00
already_have[i] = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
}
else if(m_blockchain_storage.have_tx(results[i].hash))
{
LOG_PRINT_L2("tx " << results[i].hash << " already have transaction in blockchain");
core: don't verify range proofs multiple times
2018-10-17 16:22:27 +00:00
already_have[i] = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
}
else
{
core: remove threadpool dependency from header
2018-04-23 17:55:58 +00:00
tpool.submit(&waiter, [&, i, it] {
core: guard against exceptions in tx verification worker threads
2017-09-10 14:33:13 +00:00
try
{
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
results[i].res = handle_incoming_tx_post(*it, tvc[i], results[i].tx, results[i].hash, results[i].prefix_hash, keeped_by_block, relayed, do_not_relay);
core: guard against exceptions in tx verification worker threads
2017-09-10 14:33:13 +00:00
}
catch (const std::exception &e)
{
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
MERROR_VER("Exception in handle_incoming_tx_post: " << e.what());
core: guard against exceptions in tx verification worker threads
2017-09-10 14:33:13 +00:00
results[i].res = false;
}
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
});
}
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
}
threadpool: allow leaf functions to run concurrently Decrease the number of worker threads by one to account for the fact the calling thread acts as a worker thread now
2018-04-26 10:44:47 +00:00
waiter.wait(&tpool);
moved all stuff to github
2014-03-03 22:07:58 +00:00
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
std::vector<tx_verification_batch_info> tx_info;
tx_info.reserve(tx_blobs.size());
for (size_t i = 0; i < tx_blobs.size(); i++) {
core: don't verify range proofs multiple times
2018-10-17 16:22:27 +00:00
if (!results[i].res || already_have[i])
Bulletproof aggregated verification and tests Also constrains bulletproofs to simple rct, for simplicity
2018-03-30 19:29:42 +00:00
continue;
tx_info.push_back({&results[i].tx, results[i].hash, tvc[i], results[i].res});
}
if (!tx_info.empty())
handle_incoming_tx_accumulated_batch(tx_info, keeped_by_block);
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
bool ok = true;
Use a threadpool Instead of constantly creating and destroying threads
2017-09-14 03:39:37 +00:00
it = tx_blobs.begin();
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
for (size_t i = 0; i < tx_blobs.size(); i++, ++it) {
if (!results[i].res)
{
ok = false;
continue;
}
core: fix adding new pre-hoh block when a tx is already in the pool
2018-11-26 23:42:14 +00:00
if (keeped_by_block)
get_blockchain_storage().on_new_tx_from_block(results[i].tx);
if (already_have[i])
continue;
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
const size_t weight = get_transaction_weight(results[i].tx, it->size());
core: avoid unnecessary tx/blob conversions
2018-10-31 08:31:13 +00:00
ok &= add_new_tx(results[i].tx, results[i].hash, tx_blobs[i], results[i].prefix_hash, weight, tvc[i], keeped_by_block, relayed, do_not_relay);
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
if(tvc[i].m_verifivation_failed)
{MERROR_VER("Transaction verification failed: " << results[i].hash);}
else if(tvc[i].m_verifivation_impossible)
{MERROR_VER("Transaction verification impossible: " << results[i].hash);}
if(tvc[i].m_added_to_pool)
MDEBUG("tx added: " << results[i].hash);
}
return ok;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 12:01:45 +00:00
CATCH_ENTRY_L0("core::handle_incoming_txs()", false);
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
}
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_tx(const blobdata& tx_blob, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<cryptonote::blobdata> tx_blobs;
core: thread most of handle_incoming_tx
2017-07-10 14:38:56 +00:00
tx_blobs.push_back(tx_blob);
std::vector<tx_verification_context> tvcv(1);
bool r = handle_incoming_txs(tx_blobs, tvcv, keeped_by_block, relayed, do_not_relay);
tvc = tvcv[0];
moved all stuff to github
2014-03-03 22:07:58 +00:00
return r;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::get_stat_info(core_stat_info& st_inf) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
st_inf.mining_speed = m_miner.get_speed();
st_inf.alternative_blocks = m_blockchain_storage.get_alternative_blocks_count();
st_inf.blockchain_height = m_blockchain_storage.get_current_blockchain_height();
st_inf.tx_pool_size = m_mempool.get_transactions_count();
st_inf.top_block_id_str = epee::string_tools::pod_to_hex(m_blockchain_storage.get_tail_id());
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::check_tx_semantic(const transaction& tx, bool keeped_by_block) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
if(!tx.vin.size())
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx with empty inputs, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
if(!check_inputs_types_supported(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("unsupported input types for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
if(!check_outs_valid(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx with invalid outputs, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
if (tx.version > 1)
{
if (tx.rct_signatures.outPk.size() != tx.vout.size())
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx with mismatched vout/outPk count, rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
return false;
}
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(!check_money_overflow(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx has money overflow, rejected for tx id= " << get_transaction_hash(tx));
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
if (tx.version == 1)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
uint64_t amount_in = 0;
get_inputs_money_amount(tx, amount_in);
uint64_t amount_out = get_outs_money_amount(tx);
if(amount_in <= amount_out)
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx with wrong amounts: ins " << amount_in << ", outs " << amount_out << ", rejected for tx id= " << get_transaction_hash(tx));
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
return false;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
add rct to the protocol It is not yet constrained to a fork, so don't use on the real network or you'll be orphaned or rejected.
2016-06-15 22:37:13 +00:00
// for version > 1, ringct signatures check verifies amounts match
moved all stuff to github
2014-03-03 22:07:58 +00:00
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
if(!keeped_by_block && get_transaction_weight(tx) >= m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
MERROR_VER("tx is too large " << get_transaction_weight(tx) << ", expected not bigger than " << m_blockchain_storage.get_current_cumulative_block_weight_limit() - CRYPTONOTE_COINBASE_BLOB_RESERVED_SIZE);
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
//check if tx use different key images
if(!check_tx_inputs_keyimages_diff(tx))
{
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MERROR_VER("tx uses a single key image more than once");
moved all stuff to github
2014-03-03 22:07:58 +00:00
return false;
}
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 21:38:26 +00:00
if (!check_tx_inputs_ring_members_diff(tx))
{
MERROR_VER("tx uses duplicate ring members");
return false;
}
core: test key images against validity domain
2017-02-20 17:41:39 +00:00
if (!check_tx_inputs_keyimages_domain(tx))
{
MERROR_VER("tx uses key image not in the valid domain");
return false;
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::is_key_image_spent(const crypto::key_image &key_image) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 09:49:15 +00:00
{
return m_blockchain_storage.have_tx_keyimg_as_spent(key_image);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::are_key_images_spent(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
Add a is_key_image_spent daemon command and RPC call
2015-08-11 09:49:15 +00:00
{
spent.clear();
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(auto& ki: key_im)
Add a is_key_image_spent daemon command and RPC call
2015-08-11 09:49:15 +00:00
{
spent.push_back(m_blockchain_storage.have_tx_keyimg_as_spent(ki));
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_protocol: large block sync size before v4
2017-08-16 20:19:53 +00:00
size_t core::get_block_sync_size(uint64_t height) const
{
Stagenet
2018-02-16 11:04:04 +00:00
static const uint64_t quick_height = m_nettype == TESTNET ? 801219 : m_nettype == MAINNET ? 1220516 : 0;
cryptonote_protocol: large block sync size before v4
2017-08-16 20:19:53 +00:00
if (block_sync_size > 0)
return block_sync_size;
if (height >= quick_height)
return BLOCKS_SYNCHRONIZING_DEFAULT_COUNT;
return BLOCKS_SYNCHRONIZING_DEFAULT_COUNT_PRE_V4;
}
//-----------------------------------------------------------------------------------------------
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 16:20:27 +00:00
bool core::are_key_images_spent_in_pool(const std::vector<crypto::key_image>& key_im, std::vector<bool> &spent) const
{
spent.clear();
return m_mempool.check_for_key_images(key_im, spent);
}
//-----------------------------------------------------------------------------------------------
patched parameter type bug that was causing failure on 32 bit
2016-10-13 16:49:25 +00:00
std::pair<uint64_t, uint64_t> core::get_coinbase_tx_sum(const uint64_t start_offset, const size_t count)
added print_coinbase_tx_sum option
2016-10-10 19:45:51 +00:00
{
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-10 23:55:18 +00:00
uint64_t emission_amount = 0;
uint64_t total_fee_amount = 0;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 12:29:51 +00:00
if (count)
added print_coinbase_tx_sum option
2016-10-10 19:45:51 +00:00
{
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 12:29:51 +00:00
const uint64_t end = start_offset + count - 1;
m_blockchain_storage.for_blocks_range(start_offset, end,
[this, &emission_amount, &total_fee_amount](uint64_t, const crypto::hash& hash, const block& b){
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<transaction> txs;
std::vector<crypto::hash> missed_txs;
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 11:18:33 +00:00
uint64_t coinbase_amount = get_outs_money_amount(b.miner_tx);
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-10 23:55:18 +00:00
this->get_transactions(b.tx_hashes, txs, missed_txs);
core: fix recalculation of emission/fees It was counting things many, many times
2016-12-21 11:18:33 +00:00
uint64_t tx_fee_amount = 0;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(const auto& tx: txs)
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-10 23:55:18 +00:00
{
tx_fee_amount += get_tx_fee(tx);
}
emission_amount += coinbase_amount - tx_fee_amount;
total_fee_amount += tx_fee_amount;
Don't copy blockchain for coinbase_tx_sum Changed Blockchain::for_all_blocks() to for_blocks_range() Operate on blockchain in-place instead of building a copy first.
2017-06-01 12:29:51 +00:00
return true;
});
added print_coinbase_tx_sum option
2016-10-10 19:45:51 +00:00
}
print_coinbase_tx_sum now breaks output into fee and emission components
2016-10-10 23:55:18 +00:00
return std::pair<uint64_t, uint64_t>(emission_amount, total_fee_amount);
added print_coinbase_tx_sum option
2016-10-10 19:45:51 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::check_tx_inputs_keyimages_diff(const transaction& tx) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
std::unordered_set<crypto::key_image> ki;
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(const auto& in: tx.vin)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if(!ki.insert(tokey_in.k_image).second)
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: forbid duplicate ring members from v6 This avoids someone adding what amounts to mixin 0 rings in practice, as there is no other good reason to allow this.
2017-07-18 21:38:26 +00:00
bool core::check_tx_inputs_ring_members_diff(const transaction& tx) const
{
const uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
if (version >= 6)
{
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
for (size_t n = 1; n < tokey_in.key_offsets.size(); ++n)
if (tokey_in.key_offsets[n] == 0)
return false;
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: test key images against validity domain
2017-02-20 17:41:39 +00:00
bool core::check_tx_inputs_keyimages_domain(const transaction& tx) const
{
std::unordered_set<crypto::key_image> ki;
for(const auto& in: tx.vin)
{
CHECKED_GET_SPECIFIC_VARIANT(in, const txin_to_key, tokey_in, false);
if (!(rct::scalarmultKey(rct::ki2rct(tokey_in.k_image), rct::curveOrder()) == rct::identity()))
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
bool core::add_new_tx(transaction& tx, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
crypto::hash tx_hash = get_transaction_hash(tx);
crypto::hash tx_prefix_hash = get_transaction_prefix_hash(tx);
blobdata bl;
t_serializable_object_to_blob(tx, bl);
v8: per byte fee, pad bulletproofs, fixed 11 ring size
2018-07-18 21:24:53 +00:00
size_t tx_weight = get_transaction_weight(tx, bl.size());
core: avoid unnecessary tx/blob conversions
2018-10-31 08:31:13 +00:00
return add_new_tx(tx, tx_hash, bl, tx_prefix_hash, tx_weight, tvc, keeped_by_block, relayed, do_not_relay);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
size_t core::get_blockchain_total_transactions() const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_total_transactions();
}
//-----------------------------------------------------------------------------------------------
core: avoid unnecessary tx/blob conversions
2018-10-31 08:31:13 +00:00
bool core::add_new_tx(transaction& tx, const crypto::hash& tx_hash, const cryptonote::blobdata &blob, const crypto::hash& tx_prefix_hash, size_t tx_weight, tx_verification_context& tvc, bool keeped_by_block, bool relayed, bool do_not_relay)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
if(m_mempool.have_tx(tx_hash))
{
LOG_PRINT_L2("tx " << tx_hash << "already have transaction in tx_pool");
return true;
}
if(m_blockchain_storage.have_tx(tx_hash))
{
LOG_PRINT_L2("tx " << tx_hash << " already have transaction in blockchain");
return true;
}
Fix V1/V2 use of hard fork related parameters Some of it uses hardcoded height, which will need some thinking for next (voted upon) fork.
2016-01-29 15:09:17 +00:00
uint8_t version = m_blockchain_storage.get_current_hard_fork_version();
core: avoid unnecessary tx/blob conversions
2018-10-31 08:31:13 +00:00
return m_mempool.add_tx(tx, tx_hash, blob, tx_weight, tvc, keeped_by_block, relayed, do_not_relay, version);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 00:26:48 +00:00
}
//-----------------------------------------------------------------------------------------------
bool core::relay_txpool_transactions()
{
// we attempt to relay txes that should be relayed, but were not
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: don't try to re-relay an empty set of pool transactions
2017-03-12 14:51:32 +00:00
if (m_mempool.get_relayable_transactions(txs) && !txs.empty())
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 00:26:48 +00:00
{
cryptonote_connection_context fake_context = AUTO_VAL_INIT(fake_context);
tx_verification_context tvc = AUTO_VAL_INIT(tvc);
NOTIFY_NEW_TRANSACTIONS::request r;
for (auto it = txs.begin(); it != txs.end(); ++it)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
r.txs.push_back(it->second);
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 00:26:48 +00:00
}
get_protocol()->relay_transactions(r, fake_context);
m_mempool.set_relayed(txs);
}
return true;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
core: notify the txpool when transactions are relayed
2016-10-22 19:46:51 +00:00
void core::on_transaction_relayed(const cryptonote::blobdata& tx_blob)
{
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<std::pair<crypto::hash, cryptonote::blobdata>> txs;
core: notify the txpool when transactions are relayed
2016-10-22 19:46:51 +00:00
cryptonote::transaction tx;
crypto::hash tx_hash, tx_prefix_hash;
if (!parse_and_validate_tx_from_blob(tx_blob, tx, tx_hash, tx_prefix_hash))
{
Fix a few minor typos
2016-12-04 13:13:54 +00:00
LOG_ERROR("Failed to parse relayed transaction");
core: notify the txpool when transactions are relayed
2016-10-22 19:46:51 +00:00
return;
}
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
txs.push_back(std::make_pair(tx_hash, std::move(tx_blob)));
core: notify the txpool when transactions are relayed
2016-10-22 19:46:51 +00:00
m_mempool.set_relayed(txs);
}
//-----------------------------------------------------------------------------------------------
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 14:55:38 +00:00
bool core::get_block_template(block& b, const account_public_address& adr, difficulty_type& diffic, uint64_t& height, uint64_t& expected_reward, const blobdata& ex_nonce)
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Add expected total reward to RPC "getblocktemplate". Only works from V5 fork onward - returns 0 before that block.
2017-03-28 14:55:38 +00:00
return m_blockchain_storage.create_block_template(b, adr, diffic, height, expected_reward, ex_nonce);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::find_blockchain_supplement(const std::list<crypto::hash>& qblock_ids, NOTIFY_RESPONSE_CHAIN_ENTRY::request& resp) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.find_blockchain_supplement(qblock_ids, resp);
}
//-----------------------------------------------------------------------------------------------
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 19:16:00 +00:00
bool core::find_blockchain_supplement(const uint64_t req_start_block, const std::list<crypto::hash>& qblock_ids, std::vector<std::pair<std::pair<cryptonote::blobdata, crypto::hash>, std::vector<std::pair<crypto::hash, cryptonote::blobdata> > > >& blocks, uint64_t& total_height, uint64_t& start_height, bool pruned, bool get_miner_tx_hash, size_t max_count) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
rpc: rework to avoid repeated calculations in get_blocks.bin
2018-04-16 19:16:00 +00:00
return m_blockchain_storage.find_blockchain_supplement(req_start_block, qblock_ids, blocks, total_height, start_height, pruned, get_miner_tx_hash, max_count);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
Add a get_outs (fully text based) version of get_outs.bin
2016-11-22 20:00:40 +00:00
bool core::get_outs(const COMMAND_RPC_GET_OUTPUTS_BIN::request& req, COMMAND_RPC_GET_OUTPUTS_BIN::response& res) const
Fake outs set is now decided by the wallet This plugs a privacy leak from the wallet to the daemon, as the daemon could previously see what input is included as a transaction input, which the daemon hadn't previously supplied. Now, the wallet requests a particular set of outputs, including the real one. This can result in transactions that can't be accepted if the wallet happens to select too many outputs with non standard unlock times. The daemon could know this and select another output, but the wallet is blind to it. It's currently very unlikely since I don't think anything uses non default unlock times. The wallet requests more outputs than necessary so it can use spares if any of the returns outputs are still locked. If there are not enough spares to reach the desired mixin, the transaction will fail.
2016-08-02 20:48:09 +00:00
{
return m_blockchain_storage.get_outs(req, res);
}
//-----------------------------------------------------------------------------------------------
speed up get_output_distribution (and precalc common case)
2018-05-16 21:41:50 +00:00
bool core::get_output_distribution(uint64_t amount, uint64_t from_height, uint64_t to_height, uint64_t &start_height, std::vector<uint64_t> &distribution, uint64_t &base) const
add RPC to get a histogram of outputs of a given amount
2018-02-19 11:15:15 +00:00
{
speed up get_output_distribution (and precalc common case)
2018-05-16 21:41:50 +00:00
return m_blockchain_storage.get_output_distribution(amount, from_height, to_height, start_height, distribution, base);
add RPC to get a histogram of outputs of a given amount
2018-02-19 11:15:15 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, std::vector<uint64_t>& indexs) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, indexs);
}
//-----------------------------------------------------------------------------------------------
blockchain_db: speedup tx output gathering We know all the data we'll want for getblocks.bin is contiguous
2018-12-16 13:28:49 +00:00
bool core::get_tx_outputs_gindexs(const crypto::hash& tx_id, size_t n_txes, std::vector<std::vector<uint64_t>>& indexs) const
{
return m_blockchain_storage.get_tx_outputs_gindexs(tx_id, n_txes, indexs);
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 22:07:58 +00:00
void core::pause_mine()
{
m_miner.pause();
}
//-----------------------------------------------------------------------------------------------
void core::resume_mine()
{
m_miner.resume();
}
//-----------------------------------------------------------------------------------------------
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 10:25:12 +00:00
block_complete_entry get_block_complete_entry(block& b, tx_memory_pool &pool)
{
block_complete_entry bce;
bce.block = cryptonote::block_to_blob(b);
for (const auto &tx_hash: b.tx_hashes)
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
cryptonote::blobdata txblob;
CHECK_AND_ASSERT_THROW_MES(pool.get_transaction(tx_hash, txblob), "Transaction not found in pool");
bce.txs.push_back(txblob);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 10:25:12 +00:00
}
return bce;
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool core::handle_block_found(block& b)
{
block_verification_context bvc = boost::value_initialized<block_verification_context>();
m_miner.pause();
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<block_complete_entry> blocks;
core: guard against a mined block not finding all txes in the pool This can happen for several reasons, but mainly if another block was received, which took that tx off the pool.
2017-10-11 12:00:49 +00:00
try
{
blocks.push_back(get_block_complete_entry(b, m_mempool));
}
catch (const std::exception &e)
{
m_miner.resume();
return false;
}
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 10:25:12 +00:00
prepare_handle_incoming_blocks(blocks);
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_blockchain_storage.add_new_block(b, bvc);
core: call {prepare|cleanup}_handle_incoming_blocks when adding a mined block This ensures that a batch can't also be started/stopped out of sync by another thread and us getting in the middle
2017-03-19 10:25:12 +00:00
cleanup_handle_incoming_blocks(true);
moved all stuff to github
2014-03-03 22:07:58 +00:00
//anyway - update miner template
update_miner_block_template();
m_miner.resume();
CHECK_AND_ASSERT_MES(!bvc.m_verifivation_failed, false, "mined block failed verification");
if(bvc.m_added_to_main_chain)
{
cryptonote_connection_context exclude_context = boost::value_initialized<cryptonote_connection_context>();
NOTIFY_NEW_BLOCK::request arg = AUTO_VAL_INIT(arg);
arg.current_blockchain_height = m_blockchain_storage.get_current_blockchain_height();
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
std::vector<crypto::hash> missed_txs;
std::vector<cryptonote::blobdata> txs;
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
m_blockchain_storage.get_transactions_blobs(b.tx_hashes, txs, missed_txs);
moved all stuff to github
2014-03-03 22:07:58 +00:00
if(missed_txs.size() && m_blockchain_storage.get_block_id_by_height(get_block_height(b)) != get_block_hash(b))
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("Block found but, seems that reorganize just happened after that, do not relay this block");
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
Fix typos in various files
2018-03-01 11:36:19 +00:00
CHECK_AND_ASSERT_MES(txs.size() == b.tx_hashes.size() && !missed_txs.size(), false, "can't find some transactions in found block:" << get_block_hash(b) << " txs.size()=" << txs.size()
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ", b.tx_hashes.size()=" << b.tx_hashes.size() << ", missed_txs.size()" << missed_txs.size());
block_to_blob(b, arg.b.block);
//pack transactions
Replace BOOST_FOREACH with C++11 ranged for
2017-01-22 20:38:10 +00:00
for(auto& tx: txs)
blockchain_db: add "raw" blobdata getters for block and transaction This speeds up operations such as serving blocks to syncing peers
2017-01-15 16:05:55 +00:00
arg.b.txs.push_back(tx);
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_pprotocol->relay_block(arg, exclude_context);
}
return bvc.m_added_to_main_chain;
}
//-----------------------------------------------------------------------------------------------
void core::on_synchronized()
{
m_miner.on_synchronized();
}
//-----------------------------------------------------------------------------------------------
Cleanup test impact of adding safesyncmode() method
2017-08-22 13:54:18 +00:00
void core::safesyncmode(const bool onoff)
{
m_blockchain_storage.safesyncmode(onoff);
}
//-----------------------------------------------------------------------------------------------
moved all stuff to github
2014-03-03 22:07:58 +00:00
bool core::add_new_block(const block& b, block_verification_context& bvc)
{
return m_blockchain_storage.add_new_block(b, bvc);
}
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::prepare_handle_incoming_blocks(const std::vector<block_complete_entry> &blocks)
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
{
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
m_incoming_tx_lock.lock();
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
m_blockchain_storage.prepare_handle_incoming_blocks(blocks);
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::cleanup_handle_incoming_blocks(bool force_sync)
{
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 14:43:32 +00:00
bool success = false;
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
try {
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 14:43:32 +00:00
success = m_blockchain_storage.cleanup_handle_incoming_blocks(force_sync);
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
}
catch (...) {}
m_incoming_tx_lock.unlock();
Fix blockchain_import wedge on exception in cleanup_handle_incoming_blocks
2017-08-29 14:43:32 +00:00
return success;
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
//-----------------------------------------------------------------------------------------------
bool core::handle_incoming_block(const blobdata& block_blob, block_verification_context& bvc, bool update_miner_blocktemplate)
{
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 12:01:45 +00:00
TRY_ENTRY();
various changes to runtime checkpoint updating json checkpoints will be checked every 10 minutes, dns every 60. json checkpoints always enforced, dns still with flag. conflicting checkpoints is hard fail, but soft if dns enforce flag not set and dns checkpoints are wonky.
2014-09-29 20:30:47 +00:00
// load json & DNS checkpoints every 10min/hour respectively,
// and verify them with respect to what blocks we already have
CHECK_AND_ASSERT_MES(update_checkpoints(), false, "One or more checkpoints loaded from json or dns conflicted with existing checkpoints.");
moved all stuff to github
2014-03-03 22:07:58 +00:00
bvc = boost::value_initialized<block_verification_context>();
if(block_blob.size() > get_max_block_size())
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("WRONG BLOCK BLOB, too big size " << block_blob.size() << ", rejected");
moved all stuff to github
2014-03-03 22:07:58 +00:00
bvc.m_verifivation_failed = true;
return false;
}
block b = AUTO_VAL_INIT(b);
if(!parse_and_validate_block_from_blob(block_blob, b))
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("Failed to parse and validate new block");
moved all stuff to github
2014-03-03 22:07:58 +00:00
bvc.m_verifivation_failed = true;
return false;
}
add_new_block(b, bvc);
if(update_miner_blocktemplate && bvc.m_added_to_main_chain)
update_miner_block_template();
return true;
core: guard against exceptions in handle_incoming_{block,tx} When one happens, cleanup must be called or the incoming tx lock will stay locked
2017-08-29 12:01:45 +00:00
CATCH_ENTRY_L0("core::handle_incoming_block()", false);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
Fix for orphan fix (check blocksize)
2014-06-11 21:32:53 +00:00
// Used by the RPC server to check the size of an incoming
// block_blob
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::check_incoming_block_size(const blobdata& block_blob) const
Fix for orphan fix (check blocksize)
2014-06-11 21:32:53 +00:00
{
if(block_blob.size() > get_max_block_size())
{
moved non-critical warnings and errors to log level 1
2014-09-09 09:32:00 +00:00
LOG_PRINT_L1("WRONG BLOCK BLOB, too big size " << block_blob.size() << ", rejected");
Fix for orphan fix (check blocksize)
2014-06-11 21:32:53 +00:00
return false;
}
return true;
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
crypto::hash core::get_tail_id() const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_tail_id();
}
//-----------------------------------------------------------------------------------------------
protocol: pass blockchain cumulative difficulty when syncing Not used yet.
2017-08-12 14:24:39 +00:00
difficulty_type core::get_block_cumulative_difficulty(uint64_t height) const
{
return m_blockchain_storage.get_db().get_block_cumulative_difficulty(height);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
size_t core::get_pool_transactions_count() const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_mempool.get_transactions_count();
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::have_block(const crypto::hash& id) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.have_block(id);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::parse_tx_from_blob(transaction& tx, crypto::hash& tx_hash, crypto::hash& tx_prefix_hash, const blobdata& blob) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return parse_and_validate_tx_from_blob(blob, tx, tx_hash, tx_prefix_hash);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::check_tx_syntax(const transaction& tx) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return true;
}
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
bool core::get_pool_transactions(std::vector<transaction>& txs, bool include_sensitive_data) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
m_mempool.get_transactions(txs, include_sensitive_data);
Use const where appropriate in tx_pool
2014-07-17 14:31:44 +00:00
return true;
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 18:03:23 +00:00
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
bool core::get_pool_transaction_hashes(std::vector<crypto::hash>& txs, bool include_sensitive_data) const
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 18:03:23 +00:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
m_mempool.get_transaction_hashes(txs, include_sensitive_data);
core, wallet: faster tx pool scanning Includes a new RPC to get tx pool hashes fast.
2017-03-22 18:03:23 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
bool core::get_pool_transaction_stats(struct txpool_stats& stats, bool include_sensitive_data) const
Speedup print_pool_stats Since we're just counting txs, there's no reason to deserialize all the blobs.
2017-05-31 18:11:56 +00:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
m_mempool.get_transaction_stats(stats, include_sensitive_data);
Speedup print_pool_stats Since we're just counting txs, there's no reason to deserialize all the blobs.
2017-05-31 18:11:56 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
bool core::get_pool_transaction(const crypto::hash &id, cryptonote::blobdata& tx) const
add lightweight block propagation ("fluffy blocks") Added a new command to the P2P protocol definitions to allow querying for support flags. Implemented handling of new support flags command in net_node. Changed for_each callback template to include support flags. Updated print_connections command to show peer support flags. Added p2p constant for signaling fluffy block support. Added get_pool_transaction function to cryptnote_core. Added new commands to cryptonote protocol for relaying fluffy blocks. Implemented handling of fluffy block command in cryptonote protocol. Enabled fluffy block support in node initial configuration. Implemented get_testnet function in cryptonote_core. Made it so that fluffy blocks only run on testnet.
2016-10-26 19:00:08 +00:00
{
return m_mempool.get_transaction(id, tx);
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
//-----------------------------------------------------------------------------------------------
Move txpool to the database Integration could go further (ie, return_tx_to_pool calls should not be needed anymore, possibly other things). poolstate.bin is now obsolete.
2017-05-14 13:06:55 +00:00
bool core::pool_has_tx(const crypto::hash &id) const
{
return m_mempool.have_tx(id);
}
//-----------------------------------------------------------------------------------------------
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
bool core::get_pool_transactions_and_spent_keys_info(std::vector<tx_info>& tx_infos, std::vector<spent_key_image_info>& key_image_infos, bool include_sensitive_data) const
Add mempool output to daemon via command and RPC This is for the "print_pool" command and "get_transaction_pool" RPC method. Add mempool's spent key images to the results.
2015-04-23 12:13:07 +00:00
{
Protect node privacy by proper filtering in restricted-mode RPC answers This patch allows to filter out sensitive information for queries that rely on the pool state, when running in restricted mode. This filtering is only applied to data sent back to RPC queries. Results of inline commands typed locally in the daemon are not affected. In practice, when running with `--restricted-rpc`: * get_transaction_pool will list relayed transactions with the fields "last relayed time" and "received time" set to zero. * get_transaction_pool will not list transaction that have do_not_relay set to true, and will not list key images that are used only for such transactions * get_transaction_pool_hashes.bin will not list such transaction * get_transaction_pool_stats will not count such transactions in any of the aggregated values that are computed The implementation does not make filtering the default, so developers should be mindful of this if they add new RPC functionality. Fixes #2590.
2017-11-08 12:06:41 +00:00
return m_mempool.get_transactions_and_spent_keys_info(tx_infos, key_image_infos, include_sensitive_data);
Add mempool output to daemon via command and RPC This is for the "print_pool" command and "get_transaction_pool" RPC method. Add mempool's spent key images to the results.
2015-04-23 12:13:07 +00:00
}
//-----------------------------------------------------------------------------------------------
json serialization for rpc-relevant monero types Structured {de-,}serialization methods for (many new) types which are used for requests or responses in the RPC. New types include RPC requests and responses, and structs which compose types within those. # Conflicts: # src/cryptonote_core/blockchain.cpp
2017-09-05 16:20:27 +00:00
bool core::get_pool_for_rpc(std::vector<cryptonote::rpc::tx_in_pool>& tx_infos, cryptonote::rpc::key_images_with_tx_hashes& key_image_infos) const
{
return m_mempool.get_pool_for_rpc(tx_infos, key_image_infos);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
bool core::get_short_chain_history(std::list<crypto::hash>& ids) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_short_chain_history(ids);
}
//-----------------------------------------------------------------------------------------------
bool core::handle_get_objects(NOTIFY_REQUEST_GET_OBJECTS::request& arg, NOTIFY_RESPONSE_GET_OBJECTS::request& rsp, cryptonote_connection_context& context)
{
return m_blockchain_storage.handle_get_objects(arg, rsp);
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
crypto::hash core::get_block_id_by_height(uint64_t height) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_blockchain_storage.get_block_id_by_height(height);
}
//-----------------------------------------------------------------------------------------------
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 12:20:55 +00:00
bool core::get_block_by_hash(const crypto::hash &h, block &blk, bool *orphan) const
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
{
rpc: fix orphan_status when getting blocks It was always set to false, even for orphan blocks
2017-01-22 12:20:55 +00:00
return m_blockchain_storage.get_block_by_hash(h, blk, orphan);
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
//-----------------------------------------------------------------------------------------------
core: add consts where appropriate
2015-09-19 10:25:57 +00:00
std::string core::print_pool(bool short_format) const
moved all stuff to github
2014-03-03 22:07:58 +00:00
{
return m_mempool.print_pool(short_format);
}
//-----------------------------------------------------------------------------------------------
bool core::update_miner_block_template()
{
m_miner.on_block_chain_update();
return true;
}
//-----------------------------------------------------------------------------------------------
bool core::on_idle()
{
if(!m_starter_message_showed)
{
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
std::string main_message;
if (m_offline)
main_message = "The daemon is running offline and will not attempt to sync to the Monero network.";
else
main_message = "The daemon will start synchronizing with the network. This may take a long time to complete.";
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MGINFO_YELLOW(ENDL << "**********************************************************************" << ENDL
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
<< main_message << ENDL
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ENDL
Small cleanup of daemon synchronization output Add period to second sentence
2017-11-27 21:27:26 +00:00
<< "You can set the level of process detailization through \"set_log <level|categories>\" command," << ENDL
<< "where <level> is between 0 (no details) and 4 (very verbose), or custom category based levels (eg, *:WARNING)." << ENDL
moved all stuff to github
2014-03-03 22:07:58 +00:00
<< ENDL
Update log messages to reflect fact we no longer need to type `exit` to save database before quitting
2017-01-23 01:03:04 +00:00
<< "Use the \"help\" command to see the list of available commands." << ENDL
Added command descriptions
2017-11-22 12:53:18 +00:00
<< "Use \"help <command>\" to see a command's documentation." << ENDL
Update log messages to reflect fact we no longer need to type `exit` to save database before quitting
2017-01-23 01:03:04 +00:00
<< "**********************************************************************" << ENDL);
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_starter_message_showed = true;
}
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 17:09:57 +00:00
m_fork_moaner.do_call(boost::bind(&core::check_fork_time, this));
Relay transactions when they linger too long in the pool The last relayed time of a transaction is maintained, and transactions will be relayed again if they are still in the pool after a certain amount of time, which increases with the transaction's age. All such transactions are resent, whether or not they originated on the local node.
2015-11-21 00:26:48 +00:00
m_txpool_auto_relayer.do_call(boost::bind(&core::relay_txpool_transactions, this));
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
m_check_updates_interval.do_call(boost::bind(&core::check_updates, this));
core: warn when free disk space is low
2017-11-11 23:25:12 +00:00
m_check_disk_space_interval.do_call(boost::bind(&core::check_disk_space, this));
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 13:31:45 +00:00
m_block_rate_interval.do_call(boost::bind(&core::check_block_rate, this));
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
m_blockchain_pruning_interval.do_call(boost::bind(&core::update_blockchain_pruning, this));
moved all stuff to github
2014-03-03 22:07:58 +00:00
m_miner.on_idle();
proper tx_pool handling from CryptoZoidberg / BBR
2014-06-15 07:48:13 +00:00
m_mempool.on_idle();
moved all stuff to github
2014-03-03 22:07:58 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 17:09:57 +00:00
bool core::check_fork_time()
{
HardFork::State state = m_blockchain_storage.get_hard_fork_state();
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
const el::Level level = el::Level::Warning;
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 17:09:57 +00:00
switch (state) {
case HardFork::LikelyForked:
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MCLOG_RED(level, "global", "**********************************************************************");
MCLOG_RED(level, "global", "Last scheduled hard fork is too far in the past.");
MCLOG_RED(level, "global", "We are most likely forked from the network. Daemon update needed now.");
MCLOG_RED(level, "global", "**********************************************************************");
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 17:09:57 +00:00
break;
case HardFork::UpdateNeeded:
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MCLOG_RED(level, "global", "**********************************************************************");
cryptonote_core: change wording of fork warning message An udpate may or may not be available now, but should be soon if not. This will prevent too many people freaking out.
2018-02-18 23:44:59 +00:00
MCLOG_RED(level, "global", "Last scheduled hard fork time shows a daemon update is needed soon.");
Change logging to easylogging++ This replaces the epee and data_loggers logging systems with a single one, and also adds filename:line and explicit severity levels. Categories may be defined, and logging severity set by category (or set of categories). epee style 0-4 log level maps to a sensible severity configuration. Log files now also rotate when reaching 100 MB. To select which logs to output, use the MONERO_LOGS environment variable, with a comma separated list of categories (globs are supported), with their requested severity level after a colon. If a log matches more than one such setting, the last one in the configuration string applies. A few examples: This one is (mostly) silent, only outputting fatal errors: MONERO_LOGS=*:FATAL This one is very verbose: MONERO_LOGS=*:TRACE This one is totally silent (logwise): MONERO_LOGS="" This one outputs all errors and warnings, except for the "verify" category, which prints just fatal errors (the verify category is used for logs about incoming transactions and blocks, and it is expected that some/many will fail to verify, hence we don't want the spam): MONERO_LOGS=*:WARNING,verify:FATAL Log levels are, in decreasing order of priority: FATAL, ERROR, WARNING, INFO, DEBUG, TRACE Subcategories may be added using prefixes and globs. This example will output net.p2p logs at the TRACE level, but all other net* logs only at INFO: MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE Logs which are intended for the user (which Monero was using a lot through epee, but really isn't a nice way to go things) should use the "global" category. There are a few helper macros for using this category, eg: MGINFO("this shows up by default") or MGINFO_RED("this is red"), to try to keep a similar look and feel for now. Existing epee log macros still exist, and map to the new log levels, but since they're used as a "user facing" UI element as much as a logging system, they often don't map well to log severities (ie, a log level 0 log may be an error, or may be something we want the user to see, such as an important info). In those cases, I tried to use the new macros. In other cases, I left the existing macros in. When modifying logs, it is probably best to switch to the new macros with explicit levels. The --log-level options and set_log commands now also accept category settings, in addition to the epee style log levels.
2017-01-01 16:34:23 +00:00
MCLOG_RED(level, "global", "**********************************************************************");
core: moan when we think an update is needed to get latest hard fork info
2015-09-13 17:09:57 +00:00
break;
default:
break;
}
return true;
}
//-----------------------------------------------------------------------------------------------
Only log an error if fork version is higher AND is not known.
2018-04-16 02:02:26 +00:00
uint8_t core::get_ideal_hard_fork_version() const
{
return get_blockchain_storage().get_ideal_hard_fork_version();
}
//-----------------------------------------------------------------------------------------------
tests: fix tests build Add get_fork_version and add_ideal_fork_version to core so cryptonote_protocol does not have to need the Blockchain class directly, as it's not in its dependencies, and add those to the fake core classes in tests too.
2017-08-10 09:10:04 +00:00
uint8_t core::get_ideal_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_ideal_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
uint8_t core::get_hard_fork_version(uint64_t height) const
{
return get_blockchain_storage().get_hard_fork_version(height);
}
//-----------------------------------------------------------------------------------------------
core: add get_earliest_ideal_height_for_version()
2018-03-20 11:30:09 +00:00
uint64_t core::get_earliest_ideal_height_for_version(uint8_t version) const
{
return get_blockchain_storage().get_earliest_ideal_height_for_version(version);
}
//-----------------------------------------------------------------------------------------------
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
bool core::check_updates()
{
updates: s/monerod/monero/ for the main download Also print download success message as global log
2017-02-22 10:28:24 +00:00
static const char software[] = "monero";
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
#ifdef BUILD_TAG
static const char buildtag[] = BOOST_PP_STRINGIZE(BUILD_TAG);
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 20:48:19 +00:00
static const char subdir[] = "cli"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
#else
static const char buildtag[] = "source";
Source updates are in a source subdirectory rather than in the same directory as the prebuilt versions
2017-09-22 20:48:19 +00:00
static const char subdir[] = "source"; // because it can never be simple
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
#endif
core: make --offline also disable DNS lookups
2017-11-30 15:35:52 +00:00
if (m_offline)
return true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
if (check_updates_level == UPDATES_DISABLED)
return true;
std::string version, hash;
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
MCDEBUG("updates", "Checking for a new " << software << " version for " << buildtag);
updates: remove testnet case It need not be any different
2017-02-22 11:19:20 +00:00
if (!tools::check_updates(software, buildtag, version, hash))
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
return false;
if (tools::vercmp(version.c_str(), MONERO_VERSION) <= 0)
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 14:31:34 +00:00
{
m_update_available = false;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
return true;
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 14:31:34 +00:00
}
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 11:11:45 +00:00
std::string url = tools::get_update_url(software, subdir, buildtag, version, true);
core: print the "new update found" message in cyan, for visibility
2017-03-04 18:29:18 +00:00
MCLOG_CYAN(el::Level::Info, "global", "Version " << version << " of " << software << " for " << buildtag << " is available: " << url << ", SHA256 hash " << hash);
rpc: add a "is an update available" flag in get_info Make it easier for a user to be told when to update
2018-06-18 14:31:34 +00:00
m_update_available = true;
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
if (check_updates_level == UPDATES_NOTIFY)
return true;
updates: we now have a user URL, and an automatic one So the user can use https, while the automatic does not have to
2017-02-22 11:11:45 +00:00
url = tools::get_update_url(software, subdir, buildtag, version, false);
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
std::string filename;
const char *slash = strrchr(url.c_str(), '/');
if (slash)
filename = slash + 1;
else
filename = std::string(software) + "-update-" + version;
boost::filesystem::path path(epee::string_tools::get_current_module_folder());
path /= filename;
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
core: make update download cancellable
2017-02-26 21:00:38 +00:00
boost::unique_lock<boost::mutex> lock(m_update_mutex);
if (m_update_download != 0)
{
MCDEBUG("updates", "Already downloading update");
return true;
}
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
crypto::hash file_hash;
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
if (!tools::sha256sum(path.string(), file_hash) || (hash != epee::string_tools::pod_to_hex(file_hash)))
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
{
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
MCDEBUG("updates", "We don't have that file already, downloading");
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
const std::string tmppath = path.string() + ".tmp";
if (epee::file_io_utils::is_file_exist(tmppath))
{
MCDEBUG("updates", "We have part of the file already, resuming download");
}
core: make update download cancellable
2017-02-26 21:00:38 +00:00
m_last_update_length = 0;
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
m_update_download = tools::download_async(tmppath, url, [this, hash, path](const std::string &tmppath, const std::string &uri, bool success) {
bool remove = false, good = true;
core: make update download cancellable
2017-02-26 21:00:38 +00:00
if (success)
{
crypto::hash file_hash;
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
if (!tools::sha256sum(tmppath, file_hash))
core: make update download cancellable
2017-02-26 21:00:38 +00:00
{
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
MCERROR("updates", "Failed to hash " << tmppath);
remove = true;
good = false;
core: make update download cancellable
2017-02-26 21:00:38 +00:00
}
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
else if (hash != epee::string_tools::pod_to_hex(file_hash))
core: make update download cancellable
2017-02-26 21:00:38 +00:00
{
MCERROR("updates", "Download from " << uri << " does not match the expected hash");
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
remove = true;
good = false;
core: make update download cancellable
2017-02-26 21:00:38 +00:00
}
}
else
{
MCERROR("updates", "Failed to download " << uri);
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
good = false;
core: make update download cancellable
2017-02-26 21:00:38 +00:00
}
boost::unique_lock<boost::mutex> lock(m_update_mutex);
m_update_download = 0;
resumption support for updates using range requests
2017-12-15 10:27:03 +00:00
if (success && !remove)
{
std::error_code e = tools::replace_file(tmppath, path.string());
if (e)
{
MCERROR("updates", "Failed to rename downloaded file");
good = false;
}
}
else if (remove)
{
if (!boost::filesystem::remove(tmppath))
{
MCERROR("updates", "Failed to remove invalid downloaded file");
good = false;
}
}
if (good)
MCLOG_CYAN(el::Level::Info, "updates", "New version downloaded to " << path.string());
core: make update download cancellable
2017-02-26 21:00:38 +00:00
}, [this](const std::string &path, const std::string &uri, size_t length, ssize_t content_length) {
if (length >= m_last_update_length + 1024 * 1024 * 10)
{
m_last_update_length = length;
MCDEBUG("updates", "Downloaded " << length << "/" << (content_length ? std::to_string(content_length) : "unknown"));
}
return true;
});
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
}
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
else
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
{
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
MCDEBUG("updates", "We already have " << path << " with expected hash");
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
}
core: make update download cancellable
2017-02-26 21:00:38 +00:00
lock.unlock();
core: updates can now be downloaded (and SHA256 hash checked)
2017-02-20 20:48:36 +00:00
if (check_updates_level == UPDATES_DOWNLOAD)
return true;
core: do not download an update we already downloaded
2017-02-21 22:23:59 +00:00
MCERROR("updates", "Download/update not implemented yet");
Optionally query moneropulse DNS records to check for updates It just checks and prints a message if there is a new version for now.
2017-02-18 16:30:29 +00:00
return true;
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-11 23:25:12 +00:00
bool core::check_disk_space()
{
uint64_t free_space = get_free_space();
if (free_space < 1ull * 1024 * 1024 * 1024) // 1 GB
{
const el::Level level = el::Level::Warning;
MCLOG_RED(level, "global", "Free space is below 1 GB on " << m_config_folder);
}
return true;
}
//-----------------------------------------------------------------------------------------------
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 13:31:45 +00:00
double factorial(unsigned int n)
{
if (n <= 1)
return 1.0;
double f = n;
while (n-- > 1)
f *= n;
return f;
}
//-----------------------------------------------------------------------------------------------
static double probability(unsigned int blocks, unsigned int expected)
{
// https://www.umass.edu/wsp/resources/poisson/#computing
return pow(expected, blocks) / (factorial(blocks) * exp(expected));
}
//-----------------------------------------------------------------------------------------------
bool core::check_block_rate()
{
if (m_offline || m_target_blockchain_height > get_current_blockchain_height())
{
MDEBUG("Not checking block rate, offline or syncing");
return true;
}
static constexpr double threshold = 1. / (864000 / DIFFICULTY_TARGET_V2); // one false positive every 10 days
const time_t now = time(NULL);
const std::vector<time_t> timestamps = m_blockchain_storage.get_last_block_timestamps(60);
core: add a few more block rate window sizes The 10 minute one will never trigger for 0 blocks, as it's still fairly likely to happen even without the actual hash rate changing much, so we add a 20 minute window, where it will (for 0 blocks) and a one hour window.
2019-02-06 19:04:47 +00:00
static const unsigned int seconds[] = { 5400, 3600, 1800, 1200, 600 };
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 13:31:45 +00:00
for (size_t n = 0; n < sizeof(seconds)/sizeof(seconds[0]); ++n)
{
unsigned int b = 0;
cryptonote_core: avoid gratuitous recalculations in check_block_rate()
2018-11-05 00:13:51 +00:00
const time_t time_boundary = now - static_cast<time_t>(seconds[n]);
for (time_t ts: timestamps) b += ts >= time_boundary;
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 13:31:45 +00:00
const double p = probability(b, seconds[n] / DIFFICULTY_TARGET_V2);
MDEBUG("blocks in the last " << seconds[n] / 60 << " minutes: " << b << " (probability " << p << ")");
if (p < threshold)
{
MWARNING("There were " << b << " blocks in the last " << seconds[n] / 60 << " minutes, there might be large hash rate changes, or we might be partitioned, cut off from the Monero network or under attack. Or it could be just sheer bad luck.");
core: add --block-rate-notify This runs a command whenever the block rate deviates too much from the expectation
2019-02-06 17:09:28 +00:00
std::shared_ptr<tools::Notify> block_rate_notify = m_block_rate_notify;
if (block_rate_notify)
{
auto expected = seconds[n] / DIFFICULTY_TARGET_V2;
block_rate_notify->notify("%t", std::to_string(seconds[n] / 60).c_str(), "%b", std::to_string(b).c_str(), "%e", std::to_string(expected).c_str(), NULL);
}
cryptonote_core: warn when the block rate deviates from expectations The warning threshold is set to allow a false positive every ten days on average.
2018-09-20 13:31:45 +00:00
break; // no need to look further
}
}
return true;
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
bool core::update_blockchain_pruning()
{
return m_blockchain_storage.update_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
bool core::check_blockchain_pruning()
{
return m_blockchain_storage.check_blockchain_pruning();
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
void core::set_target_blockchain_height(uint64_t target_blockchain_height)
{
Revert "Fix fake 'network synchronized, begin using' messages" This reverts commit 78035d2b6c9922f4cd730df0766aa74f4854ccb2. The patch doesn't work, and causes constant SYNCHRONIZED OK spam.
2016-09-25 18:08:31 +00:00
m_target_blockchain_height = target_blockchain_height;
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 20:50:13 +00:00
}
//-----------------------------------------------------------------------------------------------
** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY) Bockchain: 1. Optim: Multi-thread long-hash computation when encountering groups of blocks. 2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible. 3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible. 4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks. 5. Optim: Multi-thread signature computation whenever possible. 6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD) 7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???). 8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads). Berkeley-DB: 1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc). 2. Fix: Unable to pop blocks on reorganize due to transaction errors. 3. Patch: Large number of transaction aborts when running multi-threaded bulk queries. 4. Patch: Insufficient locks error when running full sync. 5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation. 6. Optim: Add bulk queries to get output global indices. 7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 9. Optim: Added thread-safe buffers used when multi-threading bulk queries. 10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details) 11. Mod: Added checkpoint thread and auto-remove-logs option. 12. *Now usable on 32-bit systems like RPI2. LMDB: 1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect) 2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3) 3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key 4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details) 5. Mod: Auto resize to +1GB instead of multiplier x1.5 ETC: 1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete. 2. Fix: 32-bit saturation bug when computing next difficulty on large blocks. [PENDING ISSUES] 1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization. This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD. 2. Berkeley db, possible bug "unable to allocate memory". TBD. [NEW OPTIONS] (*Currently all enabled for testing purposes) 1. --fast-block-sync arg=[0:1] (default: 1) a. 0 = Compute long hash per block (may take a while depending on CPU) b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence) 2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000) a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions. b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache. Fast - Write meta-data but defer data flush. Fastest - Defer meta-data and data flush. Sync - Flush data after nblocks_per_sync and wait. Async - Flush data after nblocks_per_sync but do not wait for the operation to finish. 3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower) Max number of threads to use when computing long-hash in groups. 4. --show-time-stats arg=[0:1] (default: 1) Show benchmark related time stats. 5. --db-auto-remove-logs arg=[0:1] (default: 1) For berkeley-db only. Auto remove logs if enabled. **Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version. At the moment, you need a full resync to use this optimized version. [PERFORMANCE COMPARISON] **Some figures are approximations only. Using a baseline machine of an i7-2600K+SSD+(with full pow computation): 1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain. 2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain. 3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain. Averate procesing times (with full pow computation): lmdb-optimized: 1. tx_ave = 2.5 ms / tx 2. block_ave = 5.87 ms / block memory-official-repo: 1. tx_ave = 8.85 ms / tx 2. block_ave = 19.68 ms / block lmdb-official-repo (0f4a036437fd41a5498ee5e74e2422ea6177aa3e) 1. tx_ave = 47.8 ms / tx 2. block_ave = 64.2 ms / block **Note: The following data denotes processing times only (does not include p2p download time) lmdb-optimized processing times (with full pow computation): 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000). 2. Laptop, Dual-core / 4-threads U4200 (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000). 3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000). lmdb-optimized processing times (with per-block-checkpoint) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with full pow computation) 1. Desktop, Quad-core / 8-threads 2600k (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000). 2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000). berkeley-db optimized processing times (with per-block-checkpoint) 1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-10 20:09:32 +00:00
uint64_t core::get_target_blockchain_height() const
{
'getinfo' daemon HTTP-RPC returns 'target_height' for progress estimations
2014-06-04 20:50:13 +00:00
return m_target_blockchain_height;
}
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 19:24:43 +00:00
//-----------------------------------------------------------------------------------------------
replace std::list with std::vector on some hot paths also use reserve where appropriate
2018-04-15 23:16:02 +00:00
uint64_t core::prevalidate_block_hashes(uint64_t height, const std::vector<crypto::hash> &hashes)
precomputed block hashes are now in blocks of N (currently 256) This shaves a lot of space off binaries
2017-09-13 20:03:04 +00:00
{
return get_blockchain_storage().prevalidate_block_hashes(height, hashes);
}
//-----------------------------------------------------------------------------------------------
core: warn when free disk space is low
2017-11-11 23:25:12 +00:00
uint64_t core::get_free_space() const
{
boost::filesystem::path path(m_config_folder);
boost::filesystem::space_info si = boost::filesystem::space(path);
return si.available;
}
//-----------------------------------------------------------------------------------------------
Pruning The blockchain prunes seven eighths of prunable tx data. This saves about two thirds of the blockchain size, while keeping the node useful as a sync source for an eighth of the blockchain. No other data is currently pruned. There are three ways to prune a blockchain: - run monerod with --prune-blockchain - run "prune_blockchain" in the monerod console - run the monero-blockchain-prune utility The first two will prune in place. Due to how LMDB works, this will not reduce the blockchain size on disk. Instead, it will mark parts of the file as free, so that future data will use that free space, causing the file to not grow until free space grows scarce. The third way will create a second database, a pruned copy of the original one. Since this is a new file, this one will be smaller than the original one. Once the database is pruned, it will stay pruned as it syncs. That is, there is no need to use --prune-blockchain again, etc.
2018-04-29 22:30:51 +00:00
uint32_t core::get_blockchain_pruning_seed() const
{
return get_blockchain_storage().get_blockchain_pruning_seed();
}
//-----------------------------------------------------------------------------------------------
bool core::prune_blockchain(uint32_t pruning_seed)
{
return get_blockchain_storage().prune_blockchain(pruning_seed);
}
//-----------------------------------------------------------------------------------------------
Add start_time to get_info methods and show uptime
2017-01-08 23:50:29 +00:00
std::time_t core::get_start_time() const
{
return start_time;
}
//-----------------------------------------------------------------------------------------------
Daemon should now exit on conflicting checkpoints If the "enforce DNS checkpoints" flag is not enabled, it should not exit if DNS checkpoints conflict with the others, but should still print a warning to the user.
2014-09-30 19:24:43 +00:00
void core::graceful_exit()
{
raise(SIGTERM);
}
moved all stuff to github
2014-03-03 22:07:58 +00:00
}
Reference in a new issue Copy permalink