Getting orphaned Blocks from the Blockchain

I'm very interested in doing some analysis of orphaned blocks, but to do so I'd have to have a good way of accessing them. Blockchain.info seems like a good place to start but I don't want to abuse their API by scraping the entire Blockchain. Another attempt was to use ABE but after parsing nearly all the blockchain from my Disk (several days of work) I am unable to find a single orphaned block, and I'm starting to fear that the satoshi client prunes old orphaned blocks, which would be a pity.

Will I be able to find orphaned blocks with ABE, and if not what alternatives are there?

cdecker

Posted 2012-07-08T20:37:00.053

Reputation: 7 878

Answers

Orphaned blocks are stored in blkxxxx.dat files forever, though each node will know about different orphaned blocks. Bitcoin will print a list of all known blocks, including orphans, to debug.log if you pass it -printblocktree.

theymos

Posted 2012-07-08T20:37:00.053

Reputation: 8 228

Uh, I didn't think about them not being stored on nodes that haven't seen them in the original broadcast. So nodes that weren't online when the orphaned blocks still had a chance of becoming legit will never see it? How long has the orphaned node to be abandoned until it is no longer announced? – cdecker – 2012-07-17T09:03:25.913

@cdecker Blocks are only relayed across the network once; nodes won't relay a block if it has already been seen or if it doesn't extend the currently-longest chain. So nodes that aren't online when a block is broadcast won't see it unless it becomes part of the main chain. – theymos – 2012-07-17T19:50:54.210

1That confirms what I was seeing. So if there is no archive that kept a log of all transactions I won't be able to reconstruct forks at all, right? Blockchain.info only has forks back to block 142257. Any idea where I can find more? – cdecker – 2012-07-18T09:11:01.963

@cdecker Here's my printblocktree output on a node that's been running for a very long time: http://www.mirrorcreator.com/files/1EB4UZDD/printblocktree.txt.bz2_links

– theymos – 2012-07-18T22:07:37.380

Regarding Abe, it should load orphan blocks from the blockchain if you use the blkfile loader... I haven't looked in detail the use of the orphan_block table but it's empty on both my db's despite the fact I do have orphan blocks in them (I can't say it's a bug yet but it sure looks fishy).

You can search orphan block by hash in Abe, but to find the hashes you need to look directly in the database. This query works for me on MySQL, using binary hash storage (hence the HEX() function); be sure to remove it if you store hashes as CHAR hex strings (used to be the default).

SELECT block_height, HEX(block_hash) FROM block b
  LEFT JOIN block_next bn ON (b.block_id = bn.block_id)
  WHERE bn.next_block_id IS NULL
  ORDER BY b.block_id DESC LIMIT 1,10;

The LIMIT 1,10 is to skip the most recent loaded block which is most likely the tip; it has no block_next_id but it's not orphan neither.

If you use the RPC loader, you can only catch orphan blocks while loading the latest blocks; it won't receive any orphan from bitcoind unless if it hasn't yet been orphaned! You may catch some using abe_loader and eventually with some code I'm still testing that will allow constantly loading mempool and blocks.

Thomas Guyot-Sionnest

Posted 2012-07-08T20:37:00.053

Reputation: 331

From what I can see, ABE works pretty much like the BlockExplorer in regards to orphaned blocks, which is to say it forgets them. It appears that only Blockchain.info actively stores orphaned blocks, and even displays them in a neat list.

The only other way to analyse orphaned blocks is to keep a history of the last couple blocks and constantly keep checking whether they changed. This is the approach I used for testing a simulated 51% attack on the TestNet in my master thesis. However, orphaned blocks don't occur all that often. You can look into a question on that, or go directly to the stats.

ThePiachu

Posted 2012-07-08T20:37:00.053

Reputation: 41 594