0
I have bitcoind and lnd set up on my Raspberry Pi according to the guide here.
Everything works great for the most part, and my node is up and running. I've been able to open a channel with another node, and other nodes have opened channel with me.
However, very often when I try to run a command against lncli the operation fails with the error [lncli] rpc error: code = Unavailable desc = grpc: the connection is unavailable. The lnd logs (sudo journalctl -f -u lnd) show:
lnd.service: Main process exited, code=exited, status=1/FAILURE
lnd.service: Unit entered failed state.
lnd.service: Failed with result 'exit-code'.
At those times, it seems like bitcoin is resyncing blocks (tail -f ./bitcoin/debug.log):
init message: Rewinding blocks...
init message: Verifying blocks...
Verifying last 6 blocks at level 3
[0%]...[16%]...[33%]...[50%]...[66%]...[83%]...[99%]...[DONE].
If I wait for the above to finish and fresh UpdateTips to start rolling in, I'm able to unlock and use lncli again.
What's actually going on here? Do I just have bad luck/timing? Does bitcoind frequently rewind blocks, causing the rpc calls to fail?
1That looks like part of the initial startup sequence; Perhaps bitcoind is crashing? Might be running out of memory or some other resource. Have you checked the debug.log for more info? – Raghav Sood – 2018-05-09T12:40:20.657
The
init message, etc. stuff is from bitcoin. Running out of memory actually sounds like a possibility, will check that out! – Donnie Thomas – 2018-05-09T12:46:36.280Looks like it should have enough memory:
free --megatotal used free shared buff/cache availableMem: 994 198 35 1 759 740Swap: 1023 24 999– Donnie Thomas – 2018-05-09T12:51:33.993@RaghavSood looked into it, and seems like the bitcoind process crashes because of db corruption:
2018-05-11 12:23:42 LevelDB read failure: Corruption: block checksum mismatch. Doesn't seem like the pi is running out of memory, though. Any ideas what might cause corruption? Shitty external USB HDD, maybe? – Donnie Thomas – 2018-05-11T12:27:23.6101It's hard to say, but there are others on this site with more experience with that kind of stuff. My best guess is that the Pi was unable to keep up at some point, and the process crashed, and corrupted some part of the db. – Raghav Sood – 2018-05-12T01:57:24.163
Yeah, I think it'll require some digging.
bitcoindis able to recover the db and continue syncing after crashing, but I'm not sure why it's crashing in the first place. Might need to check the memory and hard disk for errors. Thanks for your help! – Donnie Thomas – 2018-05-12T14:59:31.440Did you able to resolve this issue? I have the same issue..I have solved these issues by adding --rpcserver flag for the commands. – cmr – 2018-10-15T05:05:24.953
@cmr in the end I switched to an Odroid with an SSD, no more failures :) – Donnie Thomas – 2018-11-07T18:58:10.340