Commit graph

180 commits

Author SHA1 Message Date
moneromooo-monero 59a66e209a
move the rct commitments to the output_amounts database
Since these are needed at the same time as the output pubkeys,
this is a whole lot faster, and takes less space. Only outputs
of 0 amount store the commitment. When reading other outputs,
a fake commitment is regenerated on the fly. This avoids having
to rewrite the database to add space for fake commitments for
existing outputs.

This code relies on two things:

- LMDB must support fixed size records per key, rather than
per database (ie, all records on key 0 are the same size, all
records for non 0 keys are same size, but records from key 0
and non 0 keys do have different sizes).

- the commitment must be directly after the rest of the data
in outkey and output_data_t.
2016-08-28 21:29:02 +01:00
moneromooo-monero dc4aad7eb5
add rct to the protocol
It is not yet constrained to a fork, so don't use on the real network
or you'll be orphaned or rejected.
2016-08-28 21:28:37 +01:00
moneromooo-monero 211d1db762
db_lmdb: update reset for recent db changes
- we need to drop the new m_tx_indices database
- we reset the version to current version

This fixes the core tests failing to initialize.
2016-08-28 21:28:35 +01:00
moneromooo-monero eb56d0f994
blockchain_db: add functions for adding/removing/getting rct commitments 2016-08-28 21:28:11 +01:00
moneromooo-monero f88029e72a
db_lmdb: do not try to modify the database in read only mode 2016-08-26 19:33:25 +01:00
Riccardo Spagni 0faf572db8
Merge pull request #948
11dc091 Fake outs set is now decided by the wallet (moneromooo-monero)
1593553 new unlocked parameter to output_histogram (moneromooo-monero)
2016-08-11 22:43:14 +02:00
moneromooo-monero 240864f529
db_lmdb: errors dropping hf starting height db on reset are not fatal
This db is now dropped unconditionally, so may or may not be there
in the first place.
2016-08-10 10:50:17 +01:00
moneromooo-monero 1593553e03
new unlocked parameter to output_histogram
This constrains the number of instances of any amount
to the unlocked ones (as defined by the default unlock time
setting: outputs with non default unlock time are not
considered, so may be counted as unlocked even if they are
not actually unlocked).
2016-08-01 22:16:00 +01:00
moneromooo-monero 121165f222
db_lmdb: add some missing api call checks 2016-07-26 22:39:51 +01:00
moneromooo-monero d7b681cd65
remove hf_starting_height db
It's not really needed, it used to be an optimization for when
that code was not using the db and needed to recalculate things
fast on startup.
2016-07-13 21:38:34 +01:00
Riccardo Spagni 30dce45643 Merge branch 'performance' of https://github.com/LMDB/bitmonero 2016-04-29 21:41:43 +02:00
moneromooo-monero 513a658c87
add a --max-concurrency flag
It sets the max number of threads to use for a parallel job.
This is different that the number of total threads, since monero
binaries typically start a lot of them.
2016-04-28 20:33:59 +01:00
Howard Chu 2b0fa05f0d Another take on migration
Delete old indices and recreate them, rather than updating them
Maybe not quite as slow as before.
2016-04-24 17:46:50 +01:00
Howard Chu c14f9efd52 Migration
Migrate from DB version 0 to version 1 on startup
2016-04-08 03:11:05 +01:00
Howard Chu d7ea7d9a23 Merge branch 'performance' into master 2016-04-05 21:13:16 +01:00
Howard Chu 372acee723 Cleanup
drop obsolete remove_output()
fix get_output_key(global), fix crash in blockchain_dump
2016-04-05 21:05:24 +01:00
Howard Chu 591e421875 Cleanup and clarify
Try to rationalize the variable names, document usage.
2016-04-05 20:57:45 +01:00
Howard Chu b2f1c58805 Use cursors in some remove functions
Helps when they're called repeatedly in one txn
2016-04-05 20:57:31 +01:00
Howard Chu 118dd69dd5 Use DUPFIXED for block_info and output_txs
Saves another ~150MB or so on the full blockchain
2016-04-05 20:55:16 +01:00
Howard Chu 6225716f3c More outputs consolidation
Also bumped DB VERSION to 1
Another significant speedup and space savings:
Get rid of global_output_indices, remove indirection from output to keys

This is the change warptangent described on irc but never got to finish.
2016-04-05 20:55:12 +01:00
Howard Chu 7c5abdc3a3 Use DUPFIXED for output_keys
Saves another 90MB on 200000 block import.
Had to bring back compare_uint64 for this, but it's safe since
this table is always 64-bit aligned.
2016-04-05 20:54:42 +01:00
Howard Chu 8e9d8e3364 Use DUPFIXED for tx_indices
Small space savings, no measurable speedup
2016-04-05 20:54:39 +01:00
Howard Chu a12f9365f8 Use DUPFIXED for block_heights
Only a small savings...
2016-04-05 20:54:07 +01:00
Howard Chu 38c2277d6f Use DUPFIXED for spent_keys 2016-04-05 20:54:07 +01:00
warptangent bf769c32ba Add back changes from revert.
m_tx_outputs doesn't need to be changed, as it's no longer dup list.
2016-04-05 20:54:07 +01:00
warptangent 46b991b362 Use MDB_APPEND mode with two tx subdbs
This is possible on those using a tx index as a key.
2016-04-05 20:54:07 +01:00
warptangent 9aadedb1d0 Schema update: tx_indices - consolidate the tx subdbs from 5 to 3 2016-04-05 20:54:06 +01:00
warptangent a2f518aa01 Schema update: tx_indices - yet less indirection 2016-04-05 20:54:06 +01:00
warptangent 8d12a8df2c Schema update: tx_indices - improve further with less indirection 2016-04-05 20:54:06 +01:00
warptangent 7c013f66e9 Add batch warning for further review 2016-04-05 20:54:06 +01:00
warptangent ae0854a431 Schema update: tx_indices 2016-04-05 20:54:06 +01:00
Howard Chu 8d252a4214 Consolidated block info 2016-04-05 20:53:59 +01:00
warptangent 132c666f67 Update schema for "tx_outputs" to use array containing amount output indices
This speeds up wallet refresh by directly retrieving a tx's amount output indices.

It removes the indirection and walking the amount output duplicate list
for every amount in each requested tx.

"tx_outputs" is used by:
Amount output indices are needed for wallet refresh.
Global output indices are needed for removing a tx.

Both amount output indices and global output indices are now stored in
an array of 64-bit unsigned ints:

tx_outputs[<tx_hash>] -> [ <a1_oi, a1_gi, a2_oi, a2_gi, ...> ]

Previously it was:
tx_outputs[<tx_hash>] -> duplicate list of <a1_gi, a2_gi, a3_gi, ...>

The amount output list had to be walked for every amount in order to
find each amount's output index, by comparing the amount's global output
index with each one in the duplicate list until a match was found.

See also d045dfa7ce
2016-04-05 20:30:50 +01:00
Howard Chu efbdde2c66 Detect map resize failures 2016-03-30 20:37:28 +01:00
moneromooo-monero 600a3cf0c0
New RPC and daemon command to get output histogram
This is a list of existing output amounts along with the number
of outputs of that amount in the blockchain.

The daemon command takes:
- no parameters: all outputs with at least 3 instances
- one parameter: all outputs with at least that many instances
- two parameters: all outputs within that many instances

The default starts at 3 to avoid massive spamming of all dust
outputs in the blockchain, and is the current minimum mixin
requirement.

An optional vector of amounts may be passed, to request
histogram only for those outputs.
2016-03-26 21:10:43 +00:00
Riccardo Spagni 0d30b65785 Merge pull request #749
bfd4a28 Update BlockchainDB documentation (Thomas Winget)
797357e Change Doxyfile, Blockchain not blockchain_storage (Thomas Winget)
c835215 remove defunct code from cryptonote::core (Thomas Winget)
50dba6d cryptonote::core doxygen documentation (Thomas Winget)
8ac329d doxygen documentation for difficulty functions (Thomas Winget)
540a76c Move checkpoint functions into checkpoints class (Thomas Winget)
1b0c98e doxygen documentation for checkpoints.{h,cpp} (Thomas Winget)
89c24ac Remove unnecessary or defunct code (Thomas Winget)
ab0ed14 doxygen include private and static members (Thomas Winget)
3a48449 Updated documentation for blockchain.* (Thomas Winget)
2016-03-25 09:12:25 +02:00
Riccardo Spagni 01e0a69c17 Revert "Merge pull request #749"
This reverts commit 7fa63a82a1, reversing
changes made to cb6be986c3.
2016-03-25 08:42:42 +02:00
Thomas Winget bfd4a28c41
Update BlockchainDB documentation
BlockchainDB is now Doxygen-compliant and its documentation is
up-to-date with recent changes.
2016-03-24 03:40:04 -04:00
moneromooo-monero f7301c3563
Revert "Print stack trace upon exceptions"
Ain't nobody got time for link/cmake skullduggery.

This reverts commit fff238ec94.
2016-03-21 10:12:23 +00:00
Riccardo Spagni 37fbb7a8b7
Merge pull request #738
79117d4 db_lmdb: include the error codes from lmdb api in error logs (moneromooo-monero)
2016-03-20 21:29:11 +02:00
moneromooo-monero 79117d4275
db_lmdb: include the error codes from lmdb api in error logs 2016-03-20 18:06:04 +00:00
moneromooo-monero fff238ec94
Print stack trace upon exceptions
Useful for debugging users' logs
2016-03-19 21:48:36 +00:00
Howard Chu db1b2db4d5 Reduce log noise 2016-03-19 12:59:05 +00:00
Howard Chu a74348e115 Add destructor for readtxns
Only if we created the readtxn. Was missing cleanups from exceptions before.
2016-03-16 11:34:13 +00:00
Howard Chu 6b0a903177 Small cleanups
Only one return and TXN_POSTFIX_RDONLY() per function
Only log rtxn_start if the rtxn wasn't already active
2016-03-15 13:35:31 +00:00
Howard Chu 01c1512f22 More for 92dd4ec6d6
Make sure we stop the right txn too
2016-03-15 13:35:23 +00:00
Riccardo Spagni 240a50f3fb
Merge pull request #723
2abdb2c avoid some val copies (Howard Chu)
2016-03-14 22:20:42 +02:00
Howard Chu 92dd4ec6d6 Hack for read/write txn mixup
save the thread ID of the writer thread so we don't try to use
the writetxn from reader threads
2016-03-14 20:19:46 +00:00
Howard Chu 2abdb2c9fd avoid some val copies 2016-03-14 09:40:49 +00:00
Riccardo Spagni dfd0e9c97d
Merge pull request #712
66c2fc7 Need to link boost::chrono in more places now (Howard Chu)
b937a2c Use boost::thread instead of std::thread (Howard Chu)
2016-03-11 20:15:10 +02:00