2
3
I am trying to have Monit monitor my bitcoind process because it crashes a lot ever since the spamming has begun.
I have this in my monitrc file:
check process bitcoind with pidfile /home/ubuntu/.bitcoin/bitcoind.pid
start program = "/usr/bin/bitcoind" as uid "ubuntu" and gid "ubuntu"
stop program = "/usr/bin/bitcoin-cli -rpcconnect=127.0.0.1 -rpcport=8332 -rpcuser=bitcoinrpc -rpcpassword=mypasswordhere stop" as uid "ubuntu" and gid "ubuntu"
When bitcoind dies, monit does detect it, but it doesn't start it as indicated in monit.log:
[UTC Jul 11 22:26:32] error : 'bitcoind' process is not running
[UTC Jul 11 22:26:32] info : 'bitcoind' trying to restart
[UTC Jul 11 22:26:32] info : 'bitcoind' start: /usr/bin/bitcoind
[UTC Jul 11 22:27:02] error : 'bitcoind' failed to start
These are the lines from my debug.log just before the crash:
2015-07-09 19:22:26 socket recv error Connection timed out (110)
2015-07-09 19:23:30 dumpaddr thread stop
2015-07-09 19:23:30 opencon thread interrupt
2015-07-09 19:23:30 addcon thread interrupt
2015-07-09 19:23:30 msghand thread interrupt
2015-07-09 19:23:30 net thread interrupt
2015-07-09 19:23:30 Shutdown: In progress...
2015-07-09 19:23:30 StopNode()
2015-07-09 19:23:30 Shutdown: done
2015-07-09 19:27:32
This is another debug.log before a different crash:
2015-07-09 16:12:09 socket send error Broken pipe (32)
2015-07-09 16:12:39 ping timeout: 1200.192793s
2015-07-09 16:22:17 socket recv error Connection reset by peer (104)
2015-07-09 16:24:28
As you can see, there aren't any consistent messages before crashing.
If I run "/usr/bin/bitcoind" from console it starts fine. I am on ubuntu. I installed bitcoin through the package manager.
Any thoughts would be greatly appreciated. Thank you.
Fixing the underlying problem would be better than just restarting the daemon constantly. It's not crashing, you're just running out of memory. Add swap if necessary, reduce memory consumption by disabling incoming connections or the wallet. – Anonymous – 2015-07-11T23:14:05.943
I have 8 gigs of memory on this instance. The only thing it is doing is serving an in house blockchain parser that queries the latest blocks and transactions when they come in. This never happened until the spam attacks begun. Im on latest: 100200. I have monitored the memory consumption with nagios and it crashes even when it has only consumed ~5 gigs. – John Russell – 2015-07-11T23:23:09.687
1If it is crashing (and not due to memory exhaustion) you really need to look into why, not just running it in a while loop. Finding out signal the daemon is exiting with would go a very long way towards diagnosing what is happening. – Anonymous – 2015-07-11T23:25:43.723
And sometimes, I get absolutely nothing:
2015-07-09 05:30:14 – John Russell – 2015-07-11T23:28:05.017
If your process is being killed for memory exhaustion (which I think it is), then add swap or buy more memory. If you are managing to crash the RPC server somehow that is probably a bug and you should file an issue.
– Anonymous – 2015-07-11T23:31:56.570Again, this instance is running on 8 gigs of memory (AWS t2.large) and I know it is not running out of memory. I'll file an issue, but until then... maybe I could get some help on strategies to automatically restart bitcoind. – John Russell – 2015-07-11T23:33:36.787
while true; do bitcoind; doneThis is really a non-solution though, find the underlying issue rather than using a terrible mallet like restarting the daemon endlessly. – Anonymous – 2015-07-11T23:34:34.640I'm unfamilar with monit, but when monit fails to start bitcoind, does bitcoind put any errors out on STDERR? Does it write anything to debug.log when it fails to start? – Nick ODell – 2015-07-12T00:29:37.863
10.10 if no longer the latest, 0.11 is. I hope you file any bug reports without the attitude. – Jannes – 2015-07-12T23:00:44.193
Bug report has been created -> https://github.com/bitcoin/bitcoin/issues/6427
– John Russell – 2015-07-13T00:17:49.887