How does "headers first" affect BLKxxxx.dat structure?

Assuming a full node running Bitcoincore v0.10 (with txindex=1), how exactly does the new "headers first" functionality affect the structure of the blockchain data? (Specifically, the blkxxxx.dat data, though perhaps I'm unaware of other files which are ancillary to the data structure).

The README.md for the release specifies that there's no v0.9.x back-compatibility for v0.10 blockchain blk data.

Because release 0.10.0 makes use of headers-first synchronization and parallel block download (see further), the block files and databases are not backwards-compatible with older versions of Bitcoin Core or other software:

Blocks will be stored on disk out of order (in the order they are received, really), which makes it incompatible with some tools or other programs. Reindexing using earlier versions will also not work anymore as a result of this.

How then, does the 0.10 version blk data get parsed by the client? And by extension, why does "headers first" change the structure at all, given it seems to logically set up "a template" for the Tx data to sit?

Wizard Of Ozzie

Posted 2015-03-27T05:25:46.560

Reputation: 4 535

Answers

Having tested with the BlockStore of NBitcoin, the format is the same. The BLK files are just almost raw blocks (it has a small additional header)

Any stored block in this file has the network to which it belongs with its size, followed by the block data. Those blocks are pointed into by the leveldb database by their (fileId, offset).

The order changed though, I myself used the fact that those blocks were ordered in the past to build the header chain from the bitcoin folder. Such code would break now. (Now I'm connecting directly to the peer node instead of relying on the block folder)

Nicolas Dorier

Posted 2015-03-27T05:25:46.560

Reputation: 689

To the best of my understanding, it's just that the blocks are stored in a different order. Headers first synchronization makes use of parallel downloads and the blocks are downloaded (and then stored) out of order. It used to be the case in older versions that blocks were downloaded and then stored in order, so that's whey they added the comment to the README.

I believe it doesn't change the structure of how any individual block is stored, just that they are now likely to not be in the proper order.

morsecoder

Posted 2015-03-27T05:25:46.560

Reputation: 12 624

Yeah, that makes sense. I am trying to get a low-level, concrete answer on this so perhaps someone well versed in reading the Github discussions can chime in – Wizard Of Ozzie – 2015-03-27T09:49:52.677

Fair enough, I recognize this answer is a little bit high-level. But I suspect there may not be any real low-level changes. The Blk___.dat files literally have raw blocks in them, so I feel like the most you can do is store them in a different order – morsecoder – 2015-03-27T12:09:09.253

I'll report back :) I do appreciate the response mate. I'm just frustrated that so much functionality of the software - including blk, bitcoin-tx, rest functionality etc etc - is all hidden in Github. – Wizard Of Ozzie – 2015-03-28T04:17:08.097