Bitcoin Core 0.17 fails after loading too many wallets

0

I can start bitcoin core loading with 200 wallets. But when I increase it to 300 wallets, it fails. Below is the debug.log when I tried to load 300 wallets.

2018-12-17T01:36:27Z init message: Loading wallet...
2018-12-17T01:36:27Z [wallet-201.dat] nFileVersion = 170001
2018-12-17T01:36:27Z [wallet-201.dat] Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total. Unknown wallet records: 1
2018-12-17T01:36:27Z [wallet-201.dat] Wallet completed loading in               1ms
2018-12-17T01:36:27Z [wallet-201.dat] setKeyPool.size() = 2
2018-12-17T01:36:27Z [wallet-201.dat] mapWallet.size() = 0
2018-12-17T01:36:27Z [wallet-201.dat] mapAddressBook.size() = 0
2018-12-17T01:36:27Z init message: Loading wallet...
2018-12-17T01:36:27Z [wallet-202.dat] nFileVersion = 170001
2018-12-17T01:36:27Z [wallet-202.dat] Keys: 3 plaintext, 0 encrypted, 3 w/ metadata, 3 total. Unknown wallet records: 1
2018-12-17T01:36:39Z [wallet-202.dat] Wallet completed loading in           12012ms
2018-12-17T01:36:39Z init message: Rescanning...
2018-12-17T01:36:39Z [wallet-202.dat] Rescanning last 1447907 blocks (from block 0)...
2018-12-17T01:36:39Z [wallet-202.dat] Rescan started from block 1447885...
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=14930383)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=14955195)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15731604)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15152698)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=14994412)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15000180)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15237899)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15073409)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15076691)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15162221)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15175705)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15238839)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15734430)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15746232)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15755504)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15759768)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15780715)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15795130)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15814178)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15841983)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15844548)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15870120)
2018-12-17T01:36:39Z Unable to open file /root/.bitcoin/testnet3/blocks/blk00155.dat
2018-12-17T01:36:39Z ERROR: ReadBlockFromDisk: OpenBlockFile failed for CBlockDiskPos(nFile=155, nPos=15897975)
2018-12-17T01:36:39Z [wallet-202.dat] Rescan completed in               1ms
2018-12-17T01:36:39Z [wallet-202.dat] setKeyPool.size() = 2
2018-12-17T01:36:39Z [wallet-202.dat] mapWallet.size() = 0
2018-12-17T01:36:39Z [wallet-202.dat] mapAddressBook.size() = 0
2018-12-17T01:36:39Z init message: Loading wallet...
2018-12-17T01:36:51Z [wallet-203.dat] Releasing wallet
2018-12-17T01:36:51Z BerkeleyEnvironment::Flush: Flush(false)
2018-12-17T01:36:51Z BerkeleyEnvironment::Flush: Flush(false) took               0ms
2018-12-17T01:36:51Z 

************************
EXCEPTION: St13runtime_error       
BerkeleyBatch: Error 24, can't open database wallet.dat       
bitcoin in AppInit()

Charles

Posted 2018-12-17T01:39:20.830

Reputation: 25

Answers

2

You are going to run into issues when opening many wallets at the same time due to memory constraints. When a wallet is opened, some data from the wallet is stored in memory. If you have too many wallets opened, then you may run out of memory to store all of that extra data. This will impact other parts of the process too and could result in corrupt databases and corrupted wallets.

Furthermore, opening wallets means that a new file is being opened. Modern operating systems have limits on the number of open files a process can have. In your case, the error number 24 means that there are too many file opened. You will need to lookup how to increase the open file limit for your OS.

I do not recommend that you continue to try to have so many wallets open at once. Bitcoin Core now has dynamic loading and unloading of wallets, so you can load a wallet when you need it and unload it when you are done. This will let you have fewer wallets open simultaneously and you will not run into such issues.

Andrew Chow

Posted 2018-12-17T01:39:20.830

Reputation: 40 910

Thanks for the suggestion. The reason I load so many wallets at the same time is because I wanted to use bitcoin walletNotify. I've tested that walletNotify won't work if I didn't load the wallet. I guess I need to import all addresses from each wallets into 1 wallet as watch-only addresses. Is there a limit on running a wallet with too many watch-only addresses? – Charles – 2018-12-17T03:19:30.027

You will run into memory issues eventually, but you won't run into the too many open files problem. – Andrew Chow – 2018-12-17T03:23:26.593

I don't see what in the log indicates memory? What am I missing? --other things that could also contribute is that bdb has many default limits that might be hit, or just running into the file descriptors ulimit. – G. Maxwell – 2018-12-17T09:37:25.337

@G. Maxwell He hit the file descriptor limit, but I am mentioning in my answer out of memory because he's going to hit that too eventually if he raises the file descriptor limit. – Andrew Chow – 2018-12-17T16:12:39.120

It's not unreasonable for a host to have hundreds of gigabytes of ram these days, ... I'd understand you saying that he'd run out of memory if he said he wanted to load tens of millions of wallets... – G. Maxwell – 2019-01-19T23:58:40.890

Asked: 2018-12-17T01:39:20.830

Viewed: 179 times

Active: 2019-01-19T23:50:58.977