What does "Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues." mean?

4

1

My bitcoind client that runs on testnet has started returning a lot of errors today. It has been doing so for a few of hours (as of 2014-08-29 9:50 UTC).

When I call it via RPC, I get the following response on ALL requests

{
  "result":null,
  "error": {
    "code":-2,
    "message":"Safe mode: Warning: The network does not appear to fully agree! Some miners appear to be experiencing issues."},
    "id":"1"
  }
}

It doesn't seem to be downloading any new blocks. It seems that I'm not the only one who's affected. As of now, http://tbtc.blockr.io/ is also stuck on blocks that are almost 5 hours old.

The version of bitcoind I use is 0.9.2.1, and the debug log says:

2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 0, prev=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 1, prev=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf
2014-08-29 05:11:59 ProcessBlock: ORPHAN BLOCK 2, prev=000000000000256194246bdabb62d3b50e2c309217258b17c543d13ec839367c
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.
2014-08-29 05:12:00 UpdateTip: new best=00000000735de11454d7de10a71a14e9695135f59b70a4eb906b1107df842270  height=277770  log2_work=61.959615  tx=2398741  date=2014-08-29 05:20:26 progress=1.000001
2014-08-29 05:12:00 UpdateTip: new best=00000000e9cc8d1bea3894d11cd4bacf4099dac3b644cb4f39b4ba3ad89ed2f8  height=277769  log2_work=61.959615  tx=2398732  date=2014-08-29 05:00:09 progress=0.999999
2014-08-29 05:12:00 UpdateTip: new best=00000000d553cd36cc43b3d1255ec395c05e1d4ad730178cd987e960931399ae  height=277768  log2_work=61.959615  tx=2398721  date=2014-08-29 04:39:40 progress=0.999997
2014-08-29 05:12:00 UpdateTip: new best=0000000000000f46520ca51be43443905945d2db5b513f35c2283c2d95a41077  height=277767  log2_work=61.959615  tx=2398683  date=2014-08-29 04:19:31 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54  height=277766  log2_work=61.959004  tx=2398667  date=2014-08-29 04:16:35 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd  height=277767  log2_work=61.959615  tx=2398670  date=2014-08-29 04:15:08 progress=0.999995
2014-08-29 05:12:00 UpdateTip: new best=0000000000000e70d898fa12f328b9ee2d97912e9a495fc80eaeeb781096cadf  height=277768  log2_work=61.960226  tx=2398674  date=2014-08-29 04:24:20 progress=0.999996
2014-08-29 05:12:00 CheckForkWarningConditions: Warning: Large valid fork found
  forking the chain at height 277766 (000000009cf297bc2a9610af823b49fc1d98e001239e99204c3c410e1ad3fe54)
  lasting to height 277767 (0000000000000e819b253066739275290fb4274b580bf62fbd279c518e6e2ebd).
Chain state database corruption likely.

What does this message mean? How is a fork of a single block a "large valid fork"? Why is this that this seems to completely freeze bitcoind? How can I fix it?

LordOfThePigs

Posted 2014-08-29T09:55:18.020

Reputation: 275

Answers

5

When Bitcoin detects that there is something seriously wrong with the network it gets into safe mode, disabling the majority of the methods exposed via its RPC API (only the okSafeMode methods will still work). It does that by filling up the error JSON node with an error code and the the generic error message you received so that the response will not be parseable:

enter image description here

The reasons that push it into safe mode can be a large valid/invalid fork, an invalid longest chain, etc.

Try adding these lines to your bitcoin.conf

checkblocks=10
checklevel=1

then restart bitcoind as well as your app and see if the issue is resolved. If not, try reindexing the blocks on your disk (via the -reindex param).

Also, consider using the -alertnotify hook to get custom notifications and/or to shut down your bitcoin apps every time this happens.

George Kimionis

Posted 2014-08-29T09:55:18.020

Reputation: 2 824

1Restarting bitcoind was enough to get rid of the message. But I still don't understand how a fork that is 2 blocks long can be considered a "large valid fork". What's the default threshold for that. I also don't get why it can't recover from such a fork on its own. – LordOfThePigs – 2014-08-29T14:47:09.417

1Well it could also be the longest chain in an invalid state, or even a large invalid fork. Applying the above will keep your bitcoind going. – George Kimionis – 2014-08-29T14:50:49.090

And how can the longest chain (or any chain, really) be in an invalid state, doesn't bitcoind check the validity of all the blocks it receives block before adding them to its block database? – LordOfThePigs – 2014-08-29T14:56:11.227

1Also note that I didn't even need to alter the configuration of my bitcoind to get it to work again. restarting it as-is just made the error message go away. That sounds like a bug in bitcoind, doesn't it? – LordOfThePigs – 2014-08-29T14:58:06.537

1If it detects (with its local, current-version rules) an illegal transaction in the longest chain for example it will consider the chain invalid but since it's also the longest chain it gets into a logical deadlock (safe mode). Loosening up the block-checking rules (as suggested above) will allow bitcoin to bypass this. It could also be resolved by the network but in the meantime your bitcoin apps will not be able to properly interact with the network. – George Kimionis – 2014-08-29T15:11:09.400

Asked: 2014-08-29T09:55:18.020

Viewed: 1 197 times

Active: 2014-08-29T14:43:20.463