0
I've been trying to speed up my node sync by increasing the dbcache. However my node tends to crash overnight. My system is running ubuntu 18.04, 4GM of ram bitcoin core v0.18.
I've tried bitcoind -dbcache=4000, also tried setting it to 3000 and 2500. All have crashed overnight.
Any thoughts on improving sync time using this or other methods?
1
You are allocating far too much dbcache. If you allocate 4000 MB to the dbcache, you will use up all of your RAM and it will crash. Your operating system needs RAM, Bitcoin Core itself needs RAM outside of the dbcache. I would highly recommend that you do not set a dbcache at all as 4 GB is really not enough RAM to handle a larger dbcache, Bitcoin Core's other RAM usage, operating system RAM usage, and whatever else may be running on your machine.
I suggest that you start Bitcoin Core without dbcache and check what your RAM usage is (using top or Ubuntu's system monitor). See how much RAM is being used and how much is free. Then you can set a higher dbcache if you have enough free.
Bitcoin Core has a few other suggestions for reducing memory usage. You can set maxmempool=<n> to a number less than the default 300. You can reduce the maximum number of connections by setting maxconnections=<n> to a number smaller than 125. Note that that only is helpful if you happen to have a lot of incoming connections. Lastly you can reduce the number of threads being used for signature validiation by using par=<n> and setting it to a number smaller than the number of cores your CPU has.
Thanks for the advice, I have been watching my ram usage using top and started bitcoind without any additional options
bitcoind. – nakamoto_connection – 2019-07-16T21:13:30.610However, as I watch my top and log file, my load average increases steadily going to
4.17, 2.97, 1.36and my cache goes well above 300mb, it seems to hit about 380mb at which point it uses up all my system ram and my server crashes.I'm now starting with
bitcoind dbcache=100to see if this prevents the cache from growing out of control.Do you know if there is a config option to set the cache size in a file? – nakamoto_connection – 2019-07-16T21:19:29.880
The command line function is not limiting the cache size. There must be a database configuration file that has been set to too high a cache limit. Any ideas how I could find that file? – nakamoto_connection – 2019-07-16T21:30:07.507
I've tried specifying a cache size by creating a
DB_CONFIGfile in the.bitcoindirectory. – nakamoto_connection – 2019-07-16T21:40:44.427My cache continues to increase in size regardless of what I set it to, this time I tried 25 megabites, here is my log file from after the crash, it just shows a cache update:
2019-07-16T22:39:10Z UpdateTip: new best=0000000000000000000ef9baad2eda04785ee40c55788def6de7e76bc90ad5b0 height=548707 version=0x20000000 log2_work=89.96435 tx=353876079 date='2018-11-04T07:37:51Z' progress=0.870169 cache=110.8MiB(850392txo)– nakamoto_connection – 2019-07-16T22:52:05.360I'm reindexing my local blockchain copy as a debug to see if that solves the issue. It would be great if there was a node debug thread. – nakamoto_connection – 2019-07-17T00:51:47.223
Create a
bitcoin.conffile in~/.bitcoinand add the linedbcache=<n>where<n>is the size of the dbcache that you want to set. ADB_CONFIGfile won't do anything because Bitcoin Core is not expecting a file of that name so it won't read it. Bitcoin Core's configuration file isbitcoin.conf. If you set it as a command line option (i.e.-dbcache=<n>), it should work. Don't forget the dash (-). Note the conf file needs the option without the dash. – Andrew Chow – 2019-07-17T01:01:40.320I added the
dbcache=50argument, but my cache usage still increased out of control when my reindex reached around 82% completion. See the last log entry before my node crashed again:2019-07-19T13:57:25Z Pre-allocating up to position 0x400000 in rev01329.dat 2019-07-19T13:57:25Z UpdateTip: new best=0000000000000000002e63058c023a9a1de233554f28c7b21380b6c9003f36a8 height=534292 version=0x20000000 log2_work=89.340154 tx=331282217 date='2018-07-29T17:14:04Z' progress=0.818250 cache=280.3MiB(2059661txo)– nakamoto_connection – 2019-07-19T14:20:24.137So I checked my
bitcoin.conffile and found thattxindex=1, this should be set to zero by default, right? I have set it to 0 and will see what happens. – nakamoto_connection – 2019-07-19T14:22:13.857Near the beginning of each run in the debug.log file you should see a few lines that say something about how much cache is being allocated to each of the various things that need caching in Bitcoin Core. Also, don't forget to restart Core every time you modify the conf file. – Andrew Chow – 2019-07-19T14:29:04.263
It just crashed again and I noticed that my
topfunction showed that bitcoind was using 532% of CPU resources. – nakamoto_connection – 2019-07-19T14:38:26.433Here is the beginning of my last session in the log file:
2019-07-19T14:27:24Z Using wallet wallet.dat 2019-07-19T14:27:24Z BerkeleyEnvironment::Open: LogDir=/home/satoshi/.bitcoin/database ErrorFile=/home/satoshi/.bitcoin/db.log 2019-07-19T14:27:24Z Cache configuration: 2019-07-19T14:27:24Z * Using 2.0MiB for block index database 2019-07-19T14:27:24Z * Using 8.0MiB for chain state database 2019-07-19T14:27:24Z * Using 110.0MiB for in-memory UTXO set (plus up to 286.1MiB of unused mempool space)– nakamoto_connection – 2019-07-19T14:40:27.727Is something like this the solution? https://stackoverflow.com/questions/20259235/bitcoind-0-8-4-is-having-huge-memory-leak-causing-daemon-to-crash#22738428
– nakamoto_connection – 2019-07-19T14:43:03.530That is certainly an option, however it should be possible to reduce memory and cpu usage enough to avoid crashes. I updated my answer with some additional information that you might want to check out. If CPU usage is really high, then the part about threads (and reducing the number of threads) should be useful. – Andrew Chow – 2019-07-19T16:34:33.577