Commit graph

458 commits

Author SHA1 Message Date
moneromooo-monero
91aa90fc79
blockchain: ensure all blocks get their longhash precalculated
If the number of blocks to check was not a multiple of the
number of preparation threads, the last few blocks would
not be included in the threaded long hash calculation.
Those would still get calculated when the block gets added
to the chain, however, so this was only a tiny performance
hit, rather than a security bug.
2017-07-27 18:18:49 +01:00
moneromooo-monero
ff4bcaed81
blockchain: pass correct height to get_block_longhash 2017-07-27 18:18:20 +01:00
moneromooo-monero
53af9768ce
blockchain: one off warning when seeing a block with unknown version 2017-07-24 15:25:46 +01:00
moneromooo-monero
235df7f484
blockchain_db: add a txpool tx getter which returns existence
Avoids exception spam for the "nope, not found" case
2017-06-11 15:36:48 +01:00
Howard Chu
d17c0fc2d0
Don't copy blockchain for coinbase_tx_sum
Changed Blockchain::for_all_blocks() to for_blocks_range()
Operate on blockchain in-place instead of building a copy first.
2017-06-01 14:14:24 +01:00
Riccardo Spagni
545e2b003c
Merge pull request #1982
b52abd13 Move txpool to the database (moneromooo-monero)
2017-05-30 21:12:44 +02:00
moneromooo-monero
b52abd1370
Move txpool to the database
Integration could go further (ie, return_tx_to_pool calls should
not be needed anymore, possibly other things).

poolstate.bin is now obsolete.
2017-05-25 22:23:37 +01:00
Riccardo Spagni
865bb03ecf
Merge pull request #1968
5e5b8512 Fix obsolete OpenSSL API usage (hyc)
6c72d6a0 Fix Android recognition (hyc)
e65d66fe Fix ARM64 identification (hyc)
a4673218 Clean up ARMv8-a aes_expand_key() (hyc)
a3d77901 Fix block_longhash_worker thread (hyc)
2017-04-24 10:46:23 +02:00
assylias
bff90264b8
Add expected total reward to RPC "getblocktemplate".
Only works from V5 fork onward - returns 0 before that block.
2017-04-19 18:28:16 +01:00
hyc
a3d779013d Fix block_longhash_worker thread
Wasn't getting its stack size initialized; crashes on Android
with a default stack size of 1MB.
2017-04-12 23:51:18 +01:00
Derek Zhang
2bee92dbd0 Fixed typo in v5 hard fork finalized date 2017-03-24 18:06:55 -07:00
moneromooo-monero
5a7983585e
blockchain: offset v5 addition date on testnet
to silence the update warning, since v5 was done very early
on testnet
2017-03-24 10:01:15 +00:00
moneromooo-monero
7c03349869
blockchain: lower the relay fee by 2%
This ensures we accept txes with a fee that's slightly too small,
to accomodate blockchain median jitter
2017-03-20 19:14:12 +00:00
moneromooo-monero
3396a9f2af
Add intervening v5 fork for increased min block size
Minimum mixin 4 and enforced ringct is moved from v5 to v6.
v5 is now used for an increased minimum block size (from 60000
to 300000) to cater for larger typical/minimum transaction size.

The fee algorithm is also changed to decrease the base per kB
fee, and add a cheap tier for those transactions which we do
not care if they get delayed (or even included in a block).
2017-03-15 08:32:51 +00:00
moneromooo-monero
cfa2564a40
blockchain: don't try to load an empty block hash set
If the blocks aren't being linked against a binary (such as
one of the blockchain utilities), the symbol will not be
NULL, but the size will be 0. This avoids a apurious warning
about the data hash.
2017-03-05 19:46:55 +00:00
moneromooo-monero
3153349325
core: quantize per kB fee to 8 decimals 2017-03-03 23:31:56 +00:00
moneromooo-monero
583a7b5c74
core: protect precomputed block hashes with SHA256 2017-02-23 23:18:10 +00:00
Riccardo Spagni
c3599fa7b9
update copyright year, fix occasional lack of newline at line end 2017-02-21 19:38:18 +02:00
Riccardo Spagni
49efd3add9
Merge pull request #1727
0288310e blockchain_db: add "raw" blobdata getters for block and transaction (moneromooo-monero)
2017-02-21 11:27:15 +02:00
Riccardo Spagni
59d96eedd8
Merge pull request #1724
cca95c1c blockchain_db: do not throw on expected partial results getting keys (moneromooo-monero)
2017-02-21 11:23:03 +02:00
Howard Chu
dc53e9eef2
Add a few read txns to streamline
Slight perf gain, but mainly to reduce spam at loglevel 3
2017-02-18 21:01:24 +00:00
moneromooo-monero
0288310e3b
blockchain_db: add "raw" blobdata getters for block and transaction
This speeds up operations such as serving blocks to syncing peers
2017-02-13 21:11:37 +00:00
moneromooo-monero
cca95c1c7a
blockchain_db: do not throw on expected partial results getting keys
When scanning for outputs used in a set of incoming blocks,
we expect that some of the inputs in their transactions will
not be found in the blockchain, as they could be in previous
blocks in that set. Those outputs will be scanned there at
a later point. In this case, we add a flag to control wehther
an output not being found is expected or not.
2017-02-13 19:05:30 +00:00
kenshi84
8027ce0c75 extract some basic code from libcryptonote_core into libcryptonote_basic 2017-02-08 22:45:15 +09:00
Miguel Herranz
36dd3e238f Replace BOOST_REVERSE_FOREACH with ranged for 2017-01-22 21:47:39 +01:00
moneromooo-monero
b70ab128fc
rpc: fix orphan_status when getting blocks
It was always set to false, even for orphan blocks
2017-01-22 12:20:55 +00:00
moneromooo-monero
5833d66f65
Change logging to easylogging++
This replaces the epee and data_loggers logging systems with
a single one, and also adds filename:line and explicit severity
levels. Categories may be defined, and logging severity set
by category (or set of categories). epee style 0-4 log level
maps to a sensible severity configuration. Log files now also
rotate when reaching 100 MB.

To select which logs to output, use the MONERO_LOGS environment
variable, with a comma separated list of categories (globs are
supported), with their requested severity level after a colon.
If a log matches more than one such setting, the last one in
the configuration string applies. A few examples:

This one is (mostly) silent, only outputting fatal errors:

MONERO_LOGS=*:FATAL

This one is very verbose:

MONERO_LOGS=*:TRACE

This one is totally silent (logwise):

MONERO_LOGS=""

This one outputs all errors and warnings, except for the
"verify" category, which prints just fatal errors (the verify
category is used for logs about incoming transactions and
blocks, and it is expected that some/many will fail to verify,
hence we don't want the spam):

MONERO_LOGS=*:WARNING,verify:FATAL

Log levels are, in decreasing order of priority:
FATAL, ERROR, WARNING, INFO, DEBUG, TRACE

Subcategories may be added using prefixes and globs. This
example will output net.p2p logs at the TRACE level, but all
other net* logs only at INFO:

MONERO_LOGS=*:ERROR,net*:INFO,net.p2p:TRACE

Logs which are intended for the user (which Monero was using
a lot through epee, but really isn't a nice way to go things)
should use the "global" category. There are a few helper macros
for using this category, eg: MGINFO("this shows up by default")
or MGINFO_RED("this is red"), to try to keep a similar look
and feel for now.

Existing epee log macros still exist, and map to the new log
levels, but since they're used as a "user facing" UI element
as much as a logging system, they often don't map well to log
severities (ie, a log level 0 log may be an error, or may be
something we want the user to see, such as an important info).
In those cases, I tried to use the new macros. In other cases,
I left the existing macros in. When modifying logs, it is
probably best to switch to the new macros with explicit levels.

The --log-level options and set_log commands now also accept
category settings, in addition to the epee style log levels.
2017-01-16 00:25:46 +00:00
Riccardo Spagni
99ffc49740
Merge pull request #1577
feb499aa core: check block version for alt chains too (moneromooo-monero)
2017-01-15 14:57:30 -05:00
Riccardo Spagni
1207d8ad0d
Merge pull request #1576
ba3968f6 rct: split rct checks between semantics and other (moneromooo-monero)
2017-01-15 14:56:39 -05:00
Riccardo Spagni
10c6afd316
Merge pull request #1571
81c384e4 fix do_not_relay not preventing relaying on a timer (moneromooo-monero)
2017-01-15 14:50:10 -05:00
Riccardo Spagni
51baab2112
Merge pull request #1565
50b230d1 core: ensure block size limit is set from the start (moneromooo-monero)
2017-01-15 14:47:17 -05:00
Riccardo Spagni
65e33b1bc5
Merge pull request #1506
3ff54bdd Check for correct thread before ending batch transaction (Howard Chu)
eaf8470b Must wait for previous batch to finish before starting new one (Howard Chu)
c903c554 Don't cache block height, always get from DB (Howard Chu)
eb1fb601 Tweak default db-sync-mode to fast:async:1 (Howard Chu)
0693cff9 Use batch transactions when syncing (Howard Chu)
2017-01-15 14:43:12 -05:00
Howard Chu
eaf8470b29
Must wait for previous batch to finish before starting new one 2017-01-14 22:43:06 +00:00
Howard Chu
0693cff925
Use batch transactions when syncing
Faster throughput while avoiding corruption. I.e., makes
running with --db-sync-mode safe more tolerable.
2017-01-14 22:43:06 +00:00
moneromooo-monero
feb499aaae
core: check block version for alt chains too
This is incompatible with block version voting
2017-01-14 22:41:14 +00:00
moneromooo-monero
ba3968f6ce
rct: split rct checks between semantics and other
Semantics can be checked early
2017-01-14 21:17:32 +00:00
moneromooo-monero
81c384e408
fix do_not_relay not preventing relaying on a timer
Also print its value when printing pool
2017-01-14 13:07:05 +00:00
moneromooo-monero
50b230d12c
core: ensure block size limit is set from the start
It can now be queried by RPC, so it needs to be set before
it is otherwise needed for consensus, even if no blocks had
to be added (ie, exit and restart quickly).
2017-01-13 21:06:14 +00:00
Riccardo Spagni
e75fd059dd
Merge pull request #1555
758c0eb7 fix time stats mixin (luigi1111)
2017-01-13 14:36:24 -05:00
luigi1111
758c0eb79f
fix time stats mixin
also add blobsize
2017-01-10 10:15:15 -06:00
kenshi84
f1dde1a429 wallet cli: print originating block heights of mixin keys when making transfer 2017-01-09 15:10:34 +09:00
Riccardo Spagni
15dcc5afd3
Merge pull request #1534
1607cb7e tx_pool: better block template filling algorithm (moneromooo-monero)
9731b4e5 rpc: add block size to GET_BLOCK_HEADER RPC (moneromooo-monero)
9188b346 rpc: add current block size to the getinfo call (moneromooo-monero)
2017-01-08 16:43:54 -08:00
Riccardo Spagni
fdb1cda965
Merge pull request #1502
591d8368 core: invalidate difficulty cache when resetting blockchain (moneromooo-monero)
2017-01-08 16:37:31 -08:00
Riccardo Spagni
23cf963332
Merge pull request #1493
0478ac68 blockchain: allow marking "tx not found" without an exception (moneromooo-monero)
2017-01-08 16:33:41 -08:00
moneromooo-monero
0478ac6848
blockchain: allow marking "tx not found" without an exception
This is a normal occurence in many cases, and there is no need
to spam the log with those when it is.
2017-01-07 20:52:17 +00:00
moneromooo-monero
1607cb7e0c
tx_pool: better block template filling algorithm
Continue filling until we reach the block size limit, or the
resulting coinbase decreases.

Also remove old sanity check on block size, which is now not
wanted anymore.
2017-01-07 12:36:22 +00:00
moneromooo-monero
591d83686e
core: invalidate difficulty cache when resetting blockchain 2016-12-25 20:38:11 +00:00
luigi1111
374762654e
add tx hash to time stats
Also miner tx hash to log level 1 (have you ever tried to find a testnet miner tx hash?)
2016-12-21 19:20:49 -06:00
Riccardo Spagni
f5176cb6e8
Merge pull request #1467
55fa0479 rpc: new function and RPC to get alternative chain info (moneromooo-monero)
2016-12-20 17:44:47 +02:00
kenshi84
2ac8007544 also use portable serializer for boost_serialization_helper.h and net_node.inl, completely adandon boost/archive/binary_oarchive.hpp 2016-12-20 13:04:19 +09:00
moneromooo-monero
55fa0479a0
rpc: new function and RPC to get alternative chain info 2016-12-17 11:28:49 +00:00
moneromooo-monero
dd144b14bc
blockchain: fix reorganizations past a hard fork boundary
After popping blocks from the old chain, the hard fork object's
notion of the current version was not in line with the new height,
causing the first blocks from the new chain to be rejected due
to a false expection of a newer version.
2016-12-13 02:13:23 +00:00
Riccardo Spagni
0f99423080
Merge pull request #1418
dfbb85b6 blockchain: fix setting non trovial alternate chain as invalid (moneromooo-monero)
2016-12-10 14:48:39 +02:00
moneromooo-monero
dfbb85b6fe
blockchain: fix setting non trovial alternate chain as invalid
The wrong iterator was being used.
Also preincrement iterators to avoid possibly invalidating them,
I'm not sure this is necessary, but let's be safe.
2016-12-08 22:26:27 +00:00
luigi1111
648ea6bef1
blockchain: bring the v4 fork height one block forward
This will ensure the early 0.10 daemons will barf at the fork
height, and not a bit later, which could be confusing.
2016-12-06 21:59:08 +00:00
Riccardo Spagni
f2ea2afb77
Merge pull request #1398
f4772bae Fix a few minor typos (Pierre Boyer)
2016-12-04 22:28:22 +02:00
Riccardo Spagni
8086379680
Merge pull request #1397
3f7d6fb5 Fix delayed exit when syncing (moneromooo-monero)
2016-12-04 22:27:41 +02:00
Riccardo Spagni
416f7fbd5f
Merge pull request #1392
204b1bff blockchain: use high bound block reward on error where appropriate (moneromooo-monero)
2016-12-04 22:23:03 +02:00
Pierre Boyer
f4772bae81 Fix a few minor typos 2016-12-04 14:13:54 +01:00
moneromooo-monero
3f7d6fb57d
Fix delayed exit when syncing 2016-12-04 12:27:45 +00:00
moneromooo-monero
204b1bff62
blockchain: use high bound block reward on error where appropriate
If the block reward to use for the fee calculation can't be
calculated (should not happen in practice), use a high bound,
so we use a fee overestimate that will be accepted by the network.
2016-12-01 16:34:09 +00:00
moneromooo-monero
8f6ec90c83
blockchain: reject invalid pubkeys from v4 2016-11-28 22:32:58 +00:00
moneromooo-monero
2c0173c722
Add a get_outs (fully text based) version of get_outs.bin 2016-11-22 20:00:40 +00:00
moneromooo-monero
e6deb8abda
rpc: add a dynamic fee estimation RPC call 2016-10-31 08:38:00 +00:00
moneromooo-monero
82dbba10d4
core: dynamic fee algorithm from ArticMine
The fee will vary based on the base reward and the current
block size limit:

fee = (R/R0) * (M0/M) * F0

R: base reward
R0: reference base reward (10 monero)
M: block size limit
M0: minimum block size limit (60000)
F0: 0.002 monero

Starts applying at v4
2016-10-31 08:37:08 +00:00
moneromooo-monero
88faec75fe
wallet: select part of the fake outs from recent outputs
25% of the outputs are selected from the last 5 days (if possible),
in order to avoid the common case of sending recently received
outputs again. 25% and 5 days are subject to review later, since
it's just a wallet level change.
2016-10-15 18:17:16 +01:00
moneromooo-monero
4038e86527
Add performance timers for ringct tx verification 2016-10-10 21:24:21 +01:00
Riccardo Spagni
a8e03344e5
Merge pull request #1177
3644aa9 blockchain: avoid using a reference to a temporary (moneromooo-monero)
2016-10-04 12:17:28 +02:00
Riccardo Spagni
895d20f901
Merge pull request #1123
0d036be blockchain: do not reject chain histories matching only genesis (moneromooo-monero)
2016-10-04 12:05:11 +02:00
moneromooo-monero
3644aa94f3
blockchain: avoid using a reference to a temporary 2016-10-03 23:20:50 +01:00
NanoAkron
8ed0d72b12
Moved logging to target functions rather than caller 2016-10-03 22:11:00 +01:00
moneromooo-monero
0d036bec26
blockchain: do not reject chain histories matching only genesis
This is fine if our local chain is so small that it is entirely
contained within the last jump in the peer's short chain history.
2016-09-24 11:19:00 +01:00
Riccardo Spagni
c2faab5e1a
fix v5 height 2016-09-18 20:13:47 +02:00
Riccardo Spagni
cebbcf0f64
fix v5 fork date description 2016-09-18 19:35:36 +02:00
Riccardo Spagni
c41098aacf
updated fork heights for v4 and v5 2016-09-18 19:18:16 +02:00
rckngOpossum
e29e8b1b99 compile errors fixed when PER_BLOCK_CHECKPOINT not defined 2016-09-15 15:55:07 -05:00
rckngOpossum
e49c161987 compile errors fixed when DEBUG_CREATE_BLOCK_TEMPLATE defined 2016-09-15 15:36:16 -05:00
Riccardo Spagni
e0d78858e4
Merge pull request #1022
b8c03a5 Remove blocks_per_sync limits (Howard Chu)
2016-09-01 11:32:27 +02:00
moneromooo-monero
6cf8ca2a7f
core: faster find_blockchain_supplement
Since this queries block heights for blocks that may or may not
exist, queries for non existing blocks would throw an exception,
and that would slow down the loop a lot. 7 seconds to go through
a 30 hash list.

Fix this by adding an optional return block height to block_exists
and using this instead. Actual errors will still throw an
exception.

This also cuts down on log exception spam.
2016-08-31 10:03:32 +01:00
Howard Chu
b8c03a5f10
Remove blocks_per_sync limits
The code used to cap at 5000 blocks per sync. It also treated 0 as 1.
Remove these checks; if specified as 0 do no periodic syncs at all.
Then the user is responsible for syncing in some external process.
2016-08-31 00:00:19 +01:00
moneromooo-monero
91ca0b4065
core: allow empty global indices for txes with empty vout 2016-08-29 09:28:32 +01:00
moneromooo-monero
887db9f936
blockchain: testnet heights for v3, v4, and v5 2016-08-28 21:30:54 +01:00
moneromooo-monero
b951bc8780
wallet: transfer_selected_rct now also selects fake outs 2016-08-28 21:30:34 +01:00
moneromooo-monero
4f887dea6e
increase minimum mixin to 4 on hard fork 5 2016-08-28 21:30:33 +01:00
moneromooo-monero
0815c72df7
core: allow v1 txes after HF 5 when sweeping unmixable outputs 2016-08-28 21:30:31 +01:00
moneromooo-monero
c3b3260ae5
New "Halfway RingCT" outputs for coinbase transactions
When RingCT is enabled, outputs from coinbase transactions
are created as a single output, and stored as RingCT output,
with a fake mask. Their amount is not hidden on the blockchain
itself, but they are then able to be used as fake inputs in
a RingCT ring. Since the output amounts are hidden, their
"dustiness" is not an obstacle anymore to mixing, and this
makes the coinbase transactions a lot smaller, as well as
helping the TXO set to grow more slowly.

Also add a new "Null" type of rct signature, which decreases
the size required when no signatures are to be stored, as
in a coinbase tx.
2016-08-28 21:30:26 +01:00
moneromooo-monero
93f5c625f0
rct: rework v2 txes into prunable and non prunable data
Nothing is pruned, but this allows easier changes later.
2016-08-28 21:30:18 +01:00
moneromooo-monero
d93746b6d3
rct: rework the verification preparation process
The whole rct data apart from the MLSAGs is now included in
the signed message, to avoid malleability issues.

Instead of passing the data that's not serialized as extra
parameters to the verification API, the transaction is modified
to fill all that information. This means the transaction can
not be const anymore, but it cleaner in other ways.
2016-08-28 21:30:16 +01:00
moneromooo-monero
3ab2ab3e76
rct: change the simple flag to a type
for future expansion
2016-08-28 21:30:14 +01:00
Shen Noether
c5be4b0bea
rct: avoid the need for the last II element
This element is used in the generation of the MLSAG, but isn't
needed in verification.
Also misc changes in the cryptonote code to match, by mooo.
2016-08-28 21:30:12 +01:00
moneromooo-monero
0263dd2d23
core: add some locking around pool use 2016-08-28 21:30:08 +01:00
moneromooo-monero
37bdf6ebe3
change fork settings to allow pre-rct txes for one more fork cycle 2016-08-28 21:29:50 +01:00
moneromooo-monero
cf33e1a52a
rct: do not serialize public keys in outPk
They can be reconstructed from vout
2016-08-28 21:29:43 +01:00
moneromooo-monero
a4d4d6194b
integrate simple rct api 2016-08-28 21:29:20 +01:00
moneromooo-monero
c2ec6d35b9
mixable transactions must be rct for v3 2016-08-28 21:29:07 +01:00
moneromooo-monero
59a66e209a
move the rct commitments to the output_amounts database
Since these are needed at the same time as the output pubkeys,
this is a whole lot faster, and takes less space. Only outputs
of 0 amount store the commitment. When reading other outputs,
a fake commitment is regenerated on the fly. This avoids having
to rewrite the database to add space for fake commitments for
existing outputs.

This code relies on two things:

- LMDB must support fixed size records per key, rather than
per database (ie, all records on key 0 are the same size, all
records for non 0 keys are same size, but records from key 0
and non 0 keys do have different sizes).

- the commitment must be directly after the rest of the data
in outkey and output_data_t.
2016-08-28 21:29:02 +01:00
moneromooo-monero
20e50ec7f7
ringct: do not serialize what can be reconstructed
The mixRing (output keys and commitments) and II fields (key images)
can be reconstructed from vin data.
This saves some modest amount of space in the tx.
2016-08-28 21:28:55 +01:00
moneromooo-monero
ada527558f
Use the supplied hard fork version in validate_miner_transaction
rather than using the current one. No functional changes, but may
save some bugs in the future.
2016-08-28 21:28:51 +01:00
moneromooo-monero
dc4aad7eb5
add rct to the protocol
It is not yet constrained to a fork, so don't use on the real network
or you'll be orphaned or rejected.
2016-08-28 21:28:37 +01:00
moneromooo-monero
4258dab4d6
core: new /getrandom_rctouts.bin binary RPC call
to get random ringct outputs to mix with
2016-08-28 21:28:16 +01:00
moneromooo-monero
11dc091464
Fake outs set is now decided by the wallet
This plugs a privacy leak from the wallet to the daemon,
as the daemon could previously see what input is included
as a transaction input, which the daemon hadn't previously
supplied. Now, the wallet requests a particular set of
outputs, including the real one.

This can result in transactions that can't be accepted if
the wallet happens to select too many outputs with non standard
unlock times. The daemon could know this and select another
output, but the wallet is blind to it. It's currently very
unlikely since I don't think anything uses non default
unlock times. The wallet requests more outputs than necessary
so it can use spares if any of the returns outputs are still
locked. If there are not enough spares to reach the desired
mixin, the transaction will fail.
2016-08-11 14:35:27 +01:00
moneromooo-monero
1593553e03
new unlocked parameter to output_histogram
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
2016-08-01 22:16:00 +01:00
Riccardo Spagni
30dce45643 Merge branch 'performance' of https://github.com/LMDB/bitmonero 2016-04-29 21:41:43 +02:00
moneromooo-monero
513a658c87
add a --max-concurrency flag
It sets the max number of threads to use for a parallel job.
This is different that the number of total threads, since monero
binaries typically start a lot of them.
2016-04-28 20:33:59 +01:00
moneromooo-monero
70c86561af
blockchain: add missing overflow check for already generated coins
When reaching the tail emission phase, the amount of coins will
eventually go over MONEY_SUPPLY, overflowing 64 bits. There was
a check added to blockchain_storage, but this was not ported to
the blockchain DB version.

Reported by smooth.
2016-04-17 14:46:17 +01:00
moneromooo-monero
d6fd6be5de
blockchain: update cumulative block limit when popping a block
Avoids possible issues with accepting a tx too large to fit
in an actual block.

Reported by smooth.
2016-04-17 14:46:13 +01:00
Riccardo Spagni
8f78f1407e
Merge pull request #784
087373e Fix potential race with parallel processing of txes/signatures/blocks (moneromooo-monero)
2016-04-14 16:25:53 +09:00
moneromooo-monero
4cfb4dff3e
blockchain: remove the tx validation result cache
As pointed out by smooth, a transaction's validity may change
over time as the blockchain changes.
2016-04-06 19:01:28 +01:00
Howard Chu
d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00
warptangent
8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
moneromooo-monero
087373eccf
Fix potential race with parallel processing of txes/signatures/blocks 2016-04-02 14:20:51 +01:00
Riccardo Spagni
a38ad63f8f
Merge pull request #767
24b3e90 Convey tx verification failure reasons to the RPC client (moneromooo-monero)
2016-04-02 12:02:07 +09:00
Riccardo Spagni
2b57845766
Merge pull request #765
d5d46e6 tests: obligatory hardfork unit build fix after interface change (moneromooo-monero)
25672d3 wallet: pass std::function by const ref, not value (moneromooo-monero)
0be6e08 wallet: do not leak owned amounts to the daemon unless --trusted-daemon (moneromooo-monero)
12146da wallet: change sweep_dust to sweep_unmixable (moneromooo-monero)
600a3cf New RPC and daemon command to get output histogram (moneromooo-monero)
f9a2fd2 wallet: handle rare case where fee adjustment can bump to the next kB (moneromooo-monero)
f26651a wallet: factor fee calculation (moneromooo-monero)
2016-03-27 14:09:53 +02:00
moneromooo-monero
24b3e9007a
Convey tx verification failure reasons to the RPC client
This allows appropriate action to be taken, like displaying
the reason to the user.

Do just that in simplewallet, which should help a lot in
determining why users fail to send.

Also make it so a tx which is accepted but not relayed is
seen as a success rather than a failure.
2016-03-27 12:37:18 +01:00
moneromooo-monero
600a3cf0c0
New RPC and daemon command to get output histogram
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.

The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances

The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.

An optional vector of amounts may be passed, to request
histogram only for those outputs.
2016-03-26 21:10:43 +00:00
moneromooo-monero
b8527668ff
blockchain: for v3, require miner tx to have well behaved outs
This was meant to go in v2, but the miner tx slipped through
the cracks as it doesn't go through the main tx verification
since it doesn't get added to the pool.
2016-03-25 15:11:53 +00:00
Riccardo Spagni
0d30b65785 Merge pull request #749
bfd4a28 Update BlockchainDB documentation (Thomas Winget)
797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget)
c835215 remove defunct code from cryptonote::core (Thomas Winget)
50dba6d cryptonote::core doxygen documentation (Thomas Winget)
8ac329d doxygen documentation for difficulty functions (Thomas Winget)
540a76c Move checkpoint functions into checkpoints class (Thomas Winget)
1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget)
89c24ac Remove unnecessary or defunct code (Thomas Winget)
ab0ed14 doxygen include private and static members (Thomas Winget)
3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 09:12:25 +02:00
Riccardo Spagni
01e0a69c17 Revert "Merge pull request #749"
This reverts commit 7fa63a82a1, reversing
changes made to cb6be986c3.
2016-03-25 08:42:42 +02:00
Riccardo Spagni
7fa63a82a1
Merge pull request #749
bfd4a28 Update BlockchainDB documentation (Thomas Winget)
797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget)
c835215 remove defunct code from cryptonote::core (Thomas Winget)
50dba6d cryptonote::core doxygen documentation (Thomas Winget)
8ac329d doxygen documentation for difficulty functions (Thomas Winget)
540a76c Move checkpoint functions into checkpoints class (Thomas Winget)
1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget)
89c24ac Remove unnecessary or defunct code (Thomas Winget)
ab0ed14 doxygen include private and static members (Thomas Winget)
3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 08:22:06 +02:00
Thomas Winget
540a76c5c2
Move checkpoint functions into checkpoints class
The functions in src/cryptonote_core/checkpoints_create.{h,cpp} should
be member functions of the checkpoints class, if nothing else for the
sake of keeping their documentation together.

This commit covers moving those functions to be member functions of the
checkpoints class as well as documenting those functions.
2016-03-24 03:39:41 -04:00
Thomas Winget
89c24ac2be
Remove unnecessary or defunct code 2016-03-22 20:23:06 -04:00
Thomas Winget
3a484497bc
Updated documentation for blockchain.*
All functions are now documented in doxygen format.  Comments have been
updated to reflect the current state of the code.  Many areas for
improvement in clarity and design have been noted, as well as cruft to
be removed.  These changes are not reflected in this commit both to
allow time for comment and to keep commits organized by purpose.
2016-03-22 17:30:21 -04:00
moneromooo-monero
cc4b19c32e
blockchain: fix partial block reward detection 2016-03-22 15:37:16 +00:00
Riccardo Spagni
a1c38299b3
also maybe do the block height this time, you know, just so that it actually works. 2016-03-21 13:19:49 +02:00
Riccardo Spagni
23d1538fe7
also update the timestamp for the hard fork 2016-03-21 13:09:35 +02:00
Riccardo Spagni
eda51a0dc3
set fork date for September, add hyc's GPG key, remove aabramov's 2016-03-21 12:53:49 +02:00
Howard Chu
9b3e43c327 Fix issue #706 2016-03-19 12:57:47 +00:00
Howard Chu
b937a2c915 Use boost::thread instead of std::thread
and all other associated IPC
2016-03-11 15:09:50 +00:00
warptangent
a49c355cae
Blockchain: Omit verbose time stats messages by default
This is already the default for the daemon, but by checking a command
line argument and calling a Blockchain member function setter.

Initialize the variable to false so it's not dependent on an external
command-line argument check. This allows utilities like
blockchain_import to have a reasonable default without code changes.
2016-03-03 20:14:48 -08:00
Howard Chu
5a07cefe7b Wrap some more actions in a larger read txn 2016-02-23 20:47:15 +00:00
Howard Chu
8cc7a36f0b read txn/cursor stuff
Could wrap more later.
2016-02-23 20:47:15 +00:00
moneromooo-monero
c7f82ec769
blockchain: initialize m_hardfork to NULL
It can now be set by some other code, and is thus tested
2016-02-08 20:57:20 +00:00
moneromooo-monero
a333c42cde
core_tests: add tests for hard fork behaviors (MRL-0004)
We also replace the --fakechain option with an optional structure
containing details about configuration for the core/blockchain,
for test purposes. This seems more future friendly.
2016-02-08 18:52:00 +00:00
moneromooo-monero
6c8e5c3459
blockchain: reset hardfork object when resetting blockchain
Not doing so will prevent the new genesis block from being
reset if a switch past v1 had occured already.
2016-02-08 18:22:03 +00:00
warptangent
e298b14a55
Blockchain: Update comments on removing block 2016-02-08 09:28:25 -08:00
warptangent
e02577f594
Move HardFork DB update to BlockchainDB::add_block()
Ensures the database is consistent.

Also simplifes blockchain_import in that verify mode off has less to
work around.
2016-02-08 09:28:19 -08:00
warptangent
3800875406
Make HardFork object available to BlockchainDB and derived DB implementations
This will later allow the HardFork object's DB update functions to be
called when the DB transaction that persists across block add/remove is
open.
2016-02-08 09:28:17 -08:00
warptangent
8f863e742d
Blockchain: Optionally pass in HardFork object 2016-02-08 08:50:51 -08:00
moneromooo-monero
7658ac0f45
blockchain: revert handle_get_objects adding block id on tx not found
This differs from the original CN code, and there seems to be
no reason to include the block itself, if it was found
2016-02-03 21:08:03 +00:00
Thomas Winget
1642be242d minor bugfixes and refactoring
- Blockchain should store if it's running on testnet or not

- moved loading compiled-in block hashes to its own function for clarity

- on handle_get_objects, should now correctly return false if a block's
transactions are missing

- replace instances of BOOST_FOREACH with C++11 for loops in Blockchain.
2016-02-02 18:58:39 +00:00
Riccardo Spagni
569316aea3
Merge pull request #634
7fc6fa3 wallet: forbid dust altogether in output selection where appropriate (moneromooo-monero)
5e1a739 blockchain: log number of outputs available for a new tx (moneromooo-monero)
2016-01-31 15:21:45 +02:00
Riccardo Spagni
39d73d2a27
Merge pull request #631
bcac101 daemon: fix a few issues reported by valgrind (moneromooo-monero)
a7e8174 tx_pool: fix serialization of new relayed data (moneromooo-monero)
601ad76 hardfork: fix mixup in indexing variable in get_voting_info (moneromooo-monero)
444e22f blockchain: remove unused timer (moneromooo-monero)
7edfdd8 blockchain: fix m_sync_counter uninitialized variable use (moneromooo-monero)
d97582c epee: use generate_random_bytes for new random uuids (moneromooo-monero)
17c7c9c epee: remove dodgy random code that nobody uses (moneromooo-monero)
2016-01-31 15:20:18 +02:00
moneromooo-monero
5e1a7391e8
blockchain: log number of outputs available for a new tx 2016-01-31 10:59:58 +00:00
moneromooo-monero
444e22f01a
blockchain: remove unused timer 2016-01-30 19:02:33 +00:00
moneromooo-monero
7edfdd8f24
blockchain: fix m_sync_counter uninitialized variable use
It counts the number of blocks added since last zeroing
2016-01-30 19:01:43 +00:00
moneromooo-monero
bf6d1474c0
new flush_txpool command, and associated RPC call
It can flush a particular tx, or the whole pool (the RPC command
can flush a list of transactions too)
2016-01-30 13:28:26 +00:00
moneromooo-monero
3b1d7e03fc
Fix V1/V2 use of hard fork related parameters
Some of it uses hardcoded height, which will need some thinking
for next (voted upon) fork.
2016-01-29 17:21:11 +00:00
Riccardo Spagni
de03926850
updated copyright year 2015-12-31 08:39:56 +02:00
moneromooo-monero
576effe11a
blockchain: kill ioservice on scope end, rather than manually
This ensures this will be done without fail, as the error prone
matching of every return with a call to KILL_IOSERVICE leads to
hard to debug corruption when one is missing.
2015-12-30 16:08:30 +00:00
Riccardo Spagni
fd36eea6dd
Merge pull request #572
b39aae7 Tweak 45800a25e9 (hyc)
4a5a5ff blockchain: always stop the ioservice before returning (moneromooo-monero)
78b65cf db_lmdb: safety close db at exit (moneromooo-monero)
45800a2 db_lmdb: fix a strdup/delete[] mistmatch (moneromooo-monero)
2015-12-30 09:38:41 +02:00
moneromooo-monero
4a5a5ff157
blockchain: always stop the ioservice before returning
Fixes a use after free
2015-12-28 22:47:46 +00:00
moneromooo-monero
b0541a56eb
blockchain: remove obsolete containers 2015-12-26 18:01:24 +00:00
moneromooo-monero
18a5211173
blockchain: Fix height in call to on_blockchain_dec
It was a noop anyway
2015-12-26 18:00:45 +00:00
moneromooo-monero
f33a88cfc1
blockchain: fix a few block addition bugs
If the block reward was too high, the verification failed flag
was set, but the function continued. The code which was supposed
to trap this flag and return failure failed to trap it, and,
while the block was not added to the chain, the function would
return success.
The reason for avoiding returning when the block reward problem
was detected was to be able to return any transactions to the
pool if needed. This is now mooted by moving the transaction
return code to a separate function, which is now called at all
appropriate points, making the logic much simpler, and hopefully
correct now.
We also move the hard fork version check after the prev_id check,
as block which does not go on the top of the chain might not
have the expected version there, without being invalid just for
this reason.
Last, we trap the case where a block fails to be added due to
using already spent key images, to set the verification failed
flag.
2015-12-25 22:13:38 +00:00
moneromooo-monero
a9ff11c816
blockchain: fix an off by one error in unlocked time check 2015-12-25 22:12:52 +00:00
moneromooo-monero
f294be35bc
blockchain: reinstate double spending checks in check_tx_inputs
This fixes some double spending tests.
This may or may not be unneeded in normal (non test) circumstances,
to be determined later. Keeping these for now may be slower, but safer.
2015-12-25 22:11:21 +00:00
moneromooo-monero
737b6d6cf5
blockchain: make some flag twiddling code closer to the original
Probably paranoid and unnecessary
2015-12-25 22:10:27 +00:00
moneromooo-monero
81cb0fcdcc
blockchain: fix bitflipping test with quantized block rewards
Block reward may now be less than the full amount allowed.
This was breaking the bitflipping test.
We now keep track of whether a block which was accepted by the core
has a lower than allowed block reward, and allow this in the test.
2015-12-25 22:07:58 +00:00
moneromooo-monero
22ddf09bea
blockchain: add missing m_tx_pool.on_blockchain_dec
It was missing in the port to DB.
This is actually a noop, so should not have functional changes.
2015-12-25 22:04:13 +00:00
moneromooo-monero
d837c0ca90
blockchain: fix switch to alternative blockchain for more than one block
When rolling over more than one block, the db height will decrease,
but the split height should be constant, as per the original code.
2015-12-25 22:02:07 +00:00
moneromooo-monero
5cec076e13
blockchain: add a missing validity check to rollback_blockchain_switching
It was present in the original code
2015-12-25 22:01:21 +00:00
moneromooo-monero
3cabdb5ef2
core: catch exceptions from get_output_key
This can happen when trying to find an amount that does not exist,
and fixes a core test.
2015-12-25 21:59:26 +00:00
moneromooo-monero
5eef64578b
db: throw when given a non txout_to_key output to add
The check was explicit in the original version, so it seems
safer to make it explicit here, especially as it is now done
implicitely in a different place, away from the original check.
2015-12-25 21:56:37 +00:00
moneromooo-monero
8ea7af1ba3
Allow the wallet to access hard fork information
And make it change behavior slightly when close/after first hard fork
2015-12-19 14:52:30 +00:00
moneromooo-monero
3f2970fadb
Add missing semicolons after log statements 2015-12-19 14:47:37 +00:00
moneromooo-monero
01e92ebdb0
replace std::auto_ptr with std::unique_ptr
The former is obsolete
2015-12-17 22:47:13 +00:00
warptangent
9079a32c07
Fix typo 2015-12-15 06:33:22 -08:00
warptangent
3796941d8e
blockchain.cpp: Change indentation from 4 to 2 spaces 2015-12-15 06:33:15 -08:00
warptangent
725acc7f17
Replace tabs with two spaces for consistency with rest of codebase
Remove trailing whitespace in same files.
2015-12-15 06:22:06 -08:00
moneromooo-monero
217792351d
Tone down a bit L0 logs during daemon sync 2015-12-14 00:36:37 +00:00
moneromooo-monero
10da0a0b7c
add a --fakechain argument for tests
The core tests use the blockchain, and reset it to be able
to add test data to it. This does not play nice with the
databases, since those will save that data without an explicit
save call.
We add a fakechain flag that the tests will set, which tells
the core and blockchain code to use a separate database, as
well as skip a few things like checkpoints and fixup, which
only make sense for real data.
2015-12-13 11:38:37 +00:00
moneromooo-monero
595893fcba
blockchain: log block (not chain) height in "BLOCK SUCCESFULLY ADDED"
This makes it log the same height as the original code, which is
less confusing when comparing behaviors.
2015-12-13 11:16:37 +00:00
moneromooo-monero
2369968dc3
blockchain: fix off by one in get_blocks 2015-12-13 11:16:07 +00:00
moneromooo-monero
a98e976f9e
blockchain_db: fixup missing key images in early DB version
Early DB versions did not store key images for inputs if the
transaction spending them had no outputs (ie, all fee). This
is not correct, as this would allow these outputs to be double
spent. This was fixed in 533acc30ed
a few months ago, but databases having synced blocks 2021612 and
685498 with a faulty version will be missing those key images
in the spent keys database. This code checks for this, and adds
those key images if they are missing.
2015-12-06 21:55:05 +00:00
moneromooo-monero
d887c18e33
hardfork: fix more major/minor issues
Also add some more tests, and rename some instances of
"version" and "add" for clarity.

NOTE: the starting height values are sometimes wrong.
I suspect this is due to the hard fork reorg code being
buggy, since they're good when syncing after the fact.
However, they're not actually used by the consensus code,
so I'm ignoring this for now, but this needs debugging.
2015-11-24 20:47:12 +00:00
moneromooo-monero
53c75ab4a0
blockchain: log versions as numbers, not characters 2015-11-23 15:53:59 +00:00
Riccardo Spagni
328636cdb3
fixed testnet fork point, added comment data back in 2015-11-22 19:40:38 +02:00
Riccardo Spagni
8fe1111b5b
update v2 testnet fork height 2015-11-21 15:36:13 +02:00
moneromooo-monero
932994c0cb
Relay transactions when they linger too long in the pool
The last relayed time of a transaction is maintained, and
transactions will be relayed again if they are still in the
pool after a certain amount of time, which increases with
the transaction's age. All such transactions are resent,
whether or not they originated on the local node.
2015-11-21 00:56:21 +00:00
Javier Smooth
baf101ef4a More changes for 2-min blocks
Use the correct block time for realtime fuzz on locktime
Use the correct block time to calculate next_difficulty on alt chains (will not work as-is with voting)
Lock unit tests to original block time for now
2015-11-13 00:37:35 -08:00
Javier Smooth
4fea1a5fe7 Adjust difficulty target (2 min) and full reward zone (60 kbytes) for block version 2 2015-11-13 00:30:45 -08:00
moneromooo-monero
4187e569d8
hardfork: allow per-fork voting thresholds
And setup the first fork to not vote
2015-11-08 13:04:41 +00:00
moneromooo-monero
4f873bcbaa
Remove some old/obsolete/unused code
git history's here if needed to get any of this back
2015-10-27 10:01:20 +00:00
Riccardo Spagni
e1c29c94ad
Merge pull request #444
ecbb732 Fix leak on real output when using a very recent output (moneromooo-monero)
2015-10-26 17:57:56 +02:00
Riccardo Spagni
db68eca0de
Merge pull request #442
43bca0d blockchain_utilities: new blockchain_dump diagnostic tool (moneromooo-monero)
5f397e4 Add functions to iterate through blocks, txes, outputs, key images (moneromooo-monero)
0a5a5e8 db_bdb: record numbers for recno databases start at 1 (moneromooo-monero)
50dfdc0 db_bdb: DB_KEYEMPTY is also not found for non-top recon fields (moneromooo-monero)
572780e blockchain_db: use the DNE exceptions where appropriate (moneromooo-monero)
2015-10-26 17:56:55 +02:00
moneromooo-monero
ecbb732faa
Fix leak on real output when using a very recent output
The wallet and the daemon applied different height considerations
when selecting outputs to use. This can leak information on which
input in a ring signature is the real one.

Found and originally fixed by smooth on Aeon.
2015-10-25 16:34:57 +00:00
moneromooo-monero
5f397e4412
Add functions to iterate through blocks, txes, outputs, key images 2015-10-25 12:36:11 +00:00
moneromooo-monero
6376627530
hardfork: switch voting to block minor version
Using major version would cause older daemons to reject those
blocks as they fail to deserialize blocks with a major version
which is not 1. There is no such restriction on the minor
version, so switching allows older daemons to coexist with
newer ones till the actual fork date, when most will hopefully
have updated already.

Also, for the same reason, we consider a vote for 0 to be a
vote for 1, since older daemons set minor version to 0.
2015-10-21 19:21:14 +01:00
moneromooo-monero
b13e7f284b
blockchain_export can now export to a blocks.dat format
Also make the number of blocks endian independant, and add
support for testnet
2015-10-17 00:11:26 +01:00
moneromooo-monero
ac90d488e7
from hard fork 2, all outputs must be decomposed
The wallet decomposes fully as of now too.
2015-10-11 13:02:55 +01:00
moneromooo-monero
90ccad1236
from hard fork 2, claim a quantized reward in coinbase
The small leftover is carried forward
2015-10-10 12:28:44 +01:00
moneromooo-monero
33affd2d17
blockchain: on hardfork 2, require mixin 2 at least if possible 2015-09-27 22:46:53 +01:00
moneromooo-monero
0a7421b607
hardfork: rescan speedup
Add a block height before which version 1 is assumed
Use DB transactions
2015-09-27 22:46:41 +01:00
moneromooo-monero
4bbf944df0
blockchain: on hardfork 2, allow miners to claim less money than allowed
So they can avoid dust if they so wish
2015-09-27 22:46:30 +01:00
moneromooo-monero
198f557d38
blockchain: use different hard fork settings for testnet and mainnet 2015-09-27 22:46:13 +01:00
moneromooo-monero
5b11a89a76
hardfork: most state now saved to the DB
There will be a delay on first load of an existing blockchain
as it gets reparsed for this state data.
2015-09-20 18:42:52 +01:00
moneromooo-monero
e546f3724a
Add an RPC call and daemon command to get info on hard fork voting 2015-09-19 16:47:48 +01:00
moneromooo-monero
d06713199e
blockchain: force a hardfork recalculation at load time
Since the state isn't actually saved anywhere, as the archive
code isn't called in the new DB version.
2015-09-19 16:47:42 +01:00
moneromooo-monero
a7177610b3
core: add consts where appropriate 2015-09-19 16:47:35 +01:00
moneromooo-monero
8ffc508cef
core: moan when we think an update is needed to get latest hard fork info 2015-09-13 18:09:57 +01:00
moneromooo-monero
f85498422d
blockchain: use the new hardfork class 2015-09-12 11:15:53 +01:00
moneromooo-monero
813e758b62
blockchain: remove obsolete call to libc srand
crypto::rand is now used for output selection
2015-08-24 21:58:19 +01:00
roman
59cc92b388 removed some gcc warnings. mainly unused variables. 2015-08-23 17:59:24 +02:00
moneromooo-monero
378d004b37
blockchain: mark two places where the new code differs from the old
And I'd like a comment from tewinget or someone else
2015-08-15 18:46:19 +01:00
moneromooo-monero
73d42a75d4
blockchain: update cumulative size after block addition
Block addition can fail, and the old code would not update the
cumulative size in that case.
2015-08-15 18:44:56 +01:00
moneromooo-monero
4a443775e8
blockchain: remove dead code 2015-08-15 18:44:31 +01:00
moneromooo-monero
3f9089a767
blockchain: do not try to add a tx the pool when it was nor taken out
This is an unintended difference from the old code. Though I don't
think it can actually happen in practice with the current take_tx
implementation.
2015-08-15 18:42:29 +01:00
moneromooo-monero
769d5ef0e6
blockchain: fix off by 1 in timestamp median calculations
The height function apparently used to return the index of
the last block, rather than the height of the chain. This now
seems to be incorrect, judging the the code, so we remove the
now wrong comment, as well as a couple +/- 1 adjustments
which now cause the median calculation to differ from the
original blockchain_storage version.
2015-08-15 12:37:23 +01:00
moneromooo-monero
35abef1b92
blockchain: remove dead code 2015-08-11 10:48:51 +01:00
moneromooo-monero
4f19e68476
blockchain: factor get_num_outpouts(amount) calls
It has to stay constant as we get the blockchain lock for the
entire function. Avoids some unnecessary DB accesses.
2015-08-09 18:14:30 +01:00
moneromooo-monero
275894cdef
blockchain: always select random outs using triangular distribution
It was only used by the older blockchain_storage.
We also move the code to the calling blockchain level, to avoid
replicating the code in every DB implementation. This also makes
the get_random_out method obsolete, and we delete it.
2015-08-09 18:07:44 +01:00
Riccardo Spagni
5a26676932
Merge pull request #343
e20a4dd blockchain: fix testnet syncing (to not use blocks.dat) (moneromooo-monero)
2015-07-18 22:59:02 +02:00
moneromooo-monero
e20a4ddc76
blockchain: fix testnet syncing (to not use blocks.dat)
These are mainnet blocks, and would cause syncing on testnet to
reject all incoming blocks.
2015-07-18 10:25:22 +01:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
70ae2ee711 Fixed threadpool bug when running on single core systems.
*Thanks to freshman for reporting bug.
2015-07-17 20:02:29 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
94ea3e8ed2 Removed on_idle() calls to Blockchain::store_blockchain() for lmdb.
Added option to cache tx-input verification results.
2015-07-15 23:20:25 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
e5d2680094 ** CHANGES ARE EXPERIMENTAL (FOR TESTING ONLY)
Bockchain:
1. Optim: Multi-thread long-hash computation when encountering groups of blocks.
2. Optim: Cache verified txs and return result from cache instead of re-checking whenever possible.
3. Optim: Preload output-keys when encoutering groups of blocks. Sort by amount and global-index before bulk querying database and multi-thread when possible.
4. Optim: Disable double spend check on block verification, double spend is already detected when trying to add blocks.
5. Optim: Multi-thread signature computation whenever possible.
6. Patch: Disable locking (recursive mutex) on called functions from check_tx_inputs which causes slowdowns (only seems to happen on ubuntu/VMs??? Reason: TBD)
7. Optim: Removed looped full-tx hash computation when retrieving transactions from pool (???).
8. Optim: Cache difficulty/timestamps (735 blocks) for next-difficulty calculations so that only 2 db reads per new block is needed when a new block arrives (instead of 1470 reads).

Berkeley-DB:
1. Fix: 32-bit data errors causing wrong output global indices and failure to send blocks to peers (etc).
2. Fix: Unable to pop blocks on reorganize due to transaction errors.
3. Patch: Large number of transaction aborts when running multi-threaded bulk queries.
4. Patch: Insufficient locks error when running full sync.
5. Patch: Incorrect db stats when returning from an immediate exit from "pop block" operation.
6. Optim: Add bulk queries to get output global indices.
7. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
8. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
9. Optim: Added thread-safe buffers used when multi-threading bulk queries.
10. Optim: Added support for nosync/write_nosync options for improved performance (*see --db-sync-mode option for details)
11. Mod: Added checkpoint thread and auto-remove-logs option.
12. *Now usable on 32-bit systems like RPI2.

LMDB:
1. Optim: Added custom comparison for 256-bit key tables (minor speed-up, TBD: get actual effect)
2. Optim: Modified output_keys table to store public_key+unlock_time+height for single transaction lookup (vs 3)
3. Optim: Used output_keys table retrieve public_keys instead of going through output_amounts->output_txs+output_indices->txs->output:public_key
4. Optim: Added support for sync/writemap options for improved performance (*see --db-sync-mode option for details)
5. Mod: Auto resize to +1GB instead of multiplier x1.5

ETC:
1. Minor optimizations for slow-hash for ARM (RPI2). Incomplete.
2. Fix: 32-bit saturation bug when computing next difficulty on large blocks.

[PENDING ISSUES]
1. Berkely db has a very slow "pop-block" operation. This is very noticeable on the RPI2 as it sometimes takes > 10 MINUTES to pop a block during reorganization.
   This does not happen very often however, most reorgs seem to take a few seconds but it possibly depends on the number of outputs present. TBD.
2. Berkeley db, possible bug "unable to allocate memory". TBD.

[NEW OPTIONS] (*Currently all enabled for testing purposes)
1. --fast-block-sync arg=[0:1] (default: 1)
	a. 0 = Compute long hash per block (may take a while depending on CPU)
	b. 1 = Skip long-hash and verify blocks based on embedded known good block hashes (faster, minimal CPU dependence)
2. --db-sync-mode arg=[[safe|fast|fastest]:[sync|async]:[nblocks_per_sync]] (default: fastest:async:1000)
	a. safe = fdatasync/fsync (or equivalent) per stored block. Very slow, but safest option to protect against power-out/crash conditions.
	b. fast/fastest = Enables asynchronous fdatasync/fsync (or equivalent). Useful for battery operated devices or STABLE systems with UPS and/or systems with battery backed write cache/solid state cache.
	Fast    - Write meta-data but defer data flush.
	Fastest - Defer meta-data and data flush.
	Sync    - Flush data after nblocks_per_sync and wait.
	Async   - Flush data after nblocks_per_sync but do not wait for the operation to finish.
3. --prep-blocks-threads arg=[n] (default: 4 or system max threads, whichever is lower)
        Max number of threads to use when computing long-hash in groups.
4. --show-time-stats arg=[0:1] (default: 1)
	Show benchmark related time stats.
5. --db-auto-remove-logs arg=[0:1] (default: 1)
	For berkeley-db only. Auto remove logs if enabled.

**Note: lmdb and berkeley-db have changes to the tables and are not compatible with official git head version.
	At the moment, you need a full resync to use this optimized version.

[PERFORMANCE COMPARISON]
**Some figures are approximations only.
Using a baseline machine of an i7-2600K+SSD+(with full pow computation):
1. The optimized lmdb/blockhain core can process blocks up to 585K for ~1.25 hours + download time, so it usually takes 2.5 hours to sync the full chain.
2. The current head with memory can process blocks up to 585K for ~4.2 hours + download time, so it usually takes 5.5 hours to sync the full chain.
3. The current head with lmdb can process blocks up to 585K for ~32 hours + download time and usually takes 36 hours to sync the full chain.

Averate procesing times (with full pow computation):
lmdb-optimized:
1. tx_ave = 2.5 ms / tx
2. block_ave = 5.87 ms / block
memory-official-repo:
1. tx_ave = 8.85 ms / tx
2. block_ave = 19.68 ms / block
lmdb-official-repo (0f4a036437)
1. tx_ave = 47.8 ms / tx
2. block_ave = 64.2 ms / block

**Note: The following data denotes processing times only (does not include p2p download time)
lmdb-optimized processing times (with full pow computation):
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 1.25 hours processing time (--db-sync-mode=fastest:async:1000).
2. Laptop,   Dual-core / 4-threads U4200  (3Mb) - 4.90 hours processing time (--db-sync-mode=fastest:async:1000).
3. Embedded, Quad-core / 4-threads Z3735F (2x1Mb) - 12.0 hours processing time (--db-sync-mode=fastest:async:1000).

lmdb-optimized processing times (with per-block-checkpoint)
1. Desktop,  Quad-core / 8-threads 2600k  (8Mb) - 10 minutes processing time (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with full pow computation)
1. Desktop, Quad-core / 8-threads 2600k  (8Mb) - 1.8 hours processing time (--db-sync-mode=fastest:async:1000).
2. RPI2. Improved from estimated 3 months(???) into 2.5 days (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).

berkeley-db optimized processing times (with per-block-checkpoint)
1. RPI2. 12-15 hours (*Need 2AMP supply + Clock:1Ghz + [usb+ssd] to achieve this speed) (--db-sync-mode=fastest:async:1000).
2015-07-15 23:20:16 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
1f83444d3d Update blockchain.cpp
Fix compilation error
2015-07-15 23:20:15 -07:00
NoodleDoodleNoodleDoodleNoodleDoodleNoo
da1d3c01de
Experimental BDB workaround optimizations 2015-07-15 21:13:42 -07:00
Riccardo Spagni
e01d32e52d
cleaning up, removing redundant files, renaming, fixing incorrect licenses 2015-05-31 13:40:18 +02:00
Thomas Winget
7b14d4a17f
Steps toward multiple dbs available -- working
There will need to be some more refactoring for these changes to be
considered complete/correct, but for now it's working.

new daemon cli argument "--db-type", works for LMDB and BerkeleyDB.

A good deal of refactoring is also present in this commit, namely
Blockchain no longer instantiates BlockchainDB, but rather is passed a
pointer to an already-instantiated BlockchainDB on init().
2015-03-25 12:09:44 -04:00
warptangent
275cbd4348
Add support for database open with flags
Add support to:
  - BlockchainDB, BlockchainLMDB
  - blockchain_import utility to open LMDB database with one or more
    LMDB flags.

Sample use:
  $ blockchain_import --database lmdb#nosync
  $ blockchain_import --database lmdb#nosync,nometasync
2015-03-16 00:26:59 -07:00
Thomas Winget
eee3ee7073
BlockchainDB implementations have names now
In order to make things more general, BlockchainDB now has get_db_name()
which should return a string with the "name" of that type of db.
This "name" will be the subfolder name that holds that db type's files
within the monero folder.

Small bugfix: blockchain_converter was not correctly appending this in
the prior hard-coded-string implementation of the subfolder data
directory concept.
2015-03-13 21:39:27 -04:00
Thomas Winget
5eab480cb1
Moved BlockchainDB into its own src/ subfolder
Ostensibly janitorial work, but should be more relevant later down the
line.  Things that depend on core cryptonote things (i.e.
    cryptonote_core) don't necessarily depend on BlockchainDB and thus
have no need to have BlockchainDB baked in with them.
2015-03-06 15:20:45 -05:00
warptangent
ce71abd0fe
Move LMDB storage to subfolder 2015-02-23 00:33:37 -08:00
warptangent
59305d3137
Blockchain: match original function declaration from blockchain_storage 2015-02-23 00:33:35 -08:00
warptangent
b88ab643ca
Fix Blockchain::get_tail_id() to set parameter to last block number instead of height
This reflects the behavior of blockchain_storage::get_tail_id().

Fixes #27 so that RPC method getlastblockheader works.
2015-02-22 10:41:41 -08:00
warptangent
8bd1983cdc
Blockchain: reflect log updates from blockchain_storage
See commit 4ba680f294
2015-02-01 19:30:20 -08:00
warptangent
7f9b070165
Blockchain: reflect log and assert updates from blockchain_storage
See commit cf5a8b1d6c
2015-02-01 19:30:14 -08:00
warptangent
70342ecada
Blockchain: reflect log level of blockchain_storage
Update to match LOG_PRINT_RED_Lx statements.
See commit cf5a8b1d6c
2015-02-01 19:29:18 -08:00
warptangent
c8d27fb38d
Blockchain: reflect assert behavior of blockchain_storage for get_tx_outputs_gindexs() 2015-02-01 19:29:03 -08:00
warptangent
d00ee784db
Update recently added log statement to fix possible null dereference
This would have been triggered if function was called without fourth
parameter and ring signature check failed.
2015-02-01 19:28:58 -08:00
Thomas Winget
acd4c369e4
Should fix std::min issues related to size_t 2015-01-19 17:39:38 -05:00
warptangent
800d9b9247
Remove code previously made unused and marked unused 2015-01-14 13:41:57 -08:00
warptangent
0840c2fd7e
Fix height assertion in Blockchain::handle_alternative_block()
It expects the total number of blocks of main chain, not last block id
(off-by-one error).

This again behaves like the same height assertion done in original
implementation in blockchain_storage::handle_alternative_block().

This allows a reorganization to proceed after an alternative block has
been added.
2015-01-11 21:23:02 -08:00
warptangent
63051bea1c
Fix comparison between main and alternate chain's cumulative
difficulty.

This fixes the continual reorganization between a main and alternate
chain, using the same two latest blocks from each.

The check that cumulative difficulty of the alternate chain is bigger
than main's was not using main's last block, but incorrectly using the
passed-in block's previous block. main_chain_cumulative_difficulty was
being used in two different ways. This has been split up to keep use
of main_chain_cumulative_difficulty consistent.
2015-01-11 21:23:02 -08:00
warptangent
909ea81067
Remove a have_block() check so alternate block can be processed
Remove have_block() check from Blockchain::handle_block_to_main_chain().
Add logging to have_block().

This allows blockchain reorganization to proceed further.

have_block() check here causes an error after a blockchain reorganize
begins with error: "Attempting to add block to main chain, but it's
already either there or in an alternate chain."

While reorganizing to become the main chain, a block in the
alternative chain would be refused due to have_block() rightfully
finding it in the alternative chain. The reorganization would end in
rollback, restoring to previous blockchain.

Original implementation didn't call it here, and it doesn't appear
necessary to be called from here in this implementation either. When
needed, it appears it's called prior to handle_block_to_main_chain().
2015-01-11 21:23:02 -08:00
warptangent
1701c26750
Use block index when obtaining block's difficulty for log statement
Use last block id, not number of blocks (off-by-one error).

Fixes error at start of blockchain reorganization: "Attempt to get
cumulative difficulty from height <XXXXXX> failed -- difficulty not in
db"
2015-01-11 19:57:46 -08:00
warptangent
4eba21fd48
Fix transfers to support mixins
Implement BlockchainLMDB::get_output_global_index()
- returns global output index for a given amount and amount output
  index.

Add information to debug statement for failed ring signature check
within Blockchain::check_tx_inputs()

Fixes bitmonerod RPC call "/getrandom_outs.bin" to return correct
output keys, used in creating a transaction with mixins.

TODO: get_output_global_index() could be refactored with part of
get_output_tx_and_index() as the latter uses the former's
functionality.  Keep track of LMDB read transaction.
2015-01-09 16:34:18 -08:00
warptangent
d045dfa7ce
Fix transfers (without mixins)
Fix Blockchain::get_tx_outputs_gindexs() to return amount output
indices.

Implement BlockchainLMDB::get_tx_amount_output_indices() and call it
from the function instead of BlockchainLMDB::get_tx_output_indices()

Previously, Blockchain::get_tx_outputs_gindexs() was instead returning
global output indices, which are internal to LMDB databases.

Allows bitmonerod RPC /get_o_indexes.bin to return the amount output
indices as expected.

Allows simplewallet refresh to set correct amount output indices for
incoming transfers. simplewallet can now construct and send valid
transactions (currently only without mixins).

This is a fix that doesn't require altering the structure of the
current LMDB databases.

TODO:

This can be done more efficiently by adding another LMDB database
(key-value table).

It's not used during regular transaction validation by bitmonerod. I
think it's currently used only or mainly by simplewallet for just its
own incoming transactions. So the current behavior is not a primary
bottleneck.

Currently, it's using the "output_amounts" database, walking through a
given amount's list of values, comparing each one to a given global
output index. The iteration number of the match is the desired result:
the amount output index. This is done for each global output index of
the transaction.

A tx's amount output indices can be stored in various other ways
allowing for faster lookup. Since a tx is only written once, there are
no special future write requirements for its list of indices.
2015-01-09 16:34:12 -08:00
Thomas Winget
14555eefd5
Fixes segfault in Blockchain::handle_alternative_block
This commit should fix the segfault in
Blockchain::handle_alternative_block, and also updates a few comments
that were either incorrect or incomplete.
2015-01-09 05:56:51 -05:00
warptangent
c5c100c69b Obtain tx hash and tx output index from amount and output offset
Fixes problem of obtaining incorrect outputs used for tx input.

Reverts to earlier intended behavior that was fixed in previous
commit's split of get_output_tx_and_index into two functions.
2015-01-04 19:39:43 -08:00
Thomas Winget
c50cd95674 Fixes a bug with getting output metadata from BlockchainDB
Thanks to moneromooo-monero for spotting the bug.
2015-01-04 19:39:43 -08:00
moneromooo-monero
1860658eec blockchain: do not append "testnet" to the data directory
It is already there (unless overridden via command line).
2015-01-04 19:39:42 -08:00
moneromooo-monero
198368b2e1 blockchain: fix wallet syncing from scratch
When the wallet syncs from the first block, it is fine to start
at the genesis block.
2015-01-04 19:39:42 -08:00
moneromooo-monero
b7270ab60e blockchain: add consts where appropriate 2015-01-04 19:39:41 -08:00
Thomas Winget
215e63b79f extraneous semicolon in Blockchain::complete_timestamps_vector
credit here:
https://bitcointalk.org/index.php?topic=583449.msg9562845#msg9562845
2015-01-04 19:39:40 -08:00
Thomas Winget
6c8b8acfe4 more blockchain height-related fixes, syncing other nodes code this time 2015-01-04 19:39:40 -08:00
Thomas Winget
26a7db38eb add new checkpointing behavior to Blockchain class 2015-01-04 19:31:20 -08:00
Thomas Winget
71b18d7166 moar bug fixes, removed debug prints 2015-01-04 19:31:19 -08:00
Thomas Winget
0915913111 BlockchainLMDB seems to be working*!
* - Well, mostly.  Haven't let it sync too far just yet.  Currently
trying to figure out the best way to deal with LMDB/mmap virtual memory
pages.
2015-01-04 19:31:19 -08:00