Format of a block key's contents in bitcoind's LevelDB?

4

1

I am trying to use bitcoind's LevelDB block index files (that are stored in blocks/index) to find the file (e.g. blk00029.dat) and byte position of specfic blocks.

Thanks to questions What is the database for? and What are the keys used in the blockchain levelDB (ie what are the key:value pairs)? I know where to look, however the contents of the database key for a block puzzles me. Even with the help of the source code I cannot figure out it's format.

For example for the block at height 100,000, which resides under database key b\x06\xe53\xfd\x1a\xda\x869\x1f?l42\x04\xb0\xd2x\xd4\xaa\xec\x1c\x0b \xaa'\xba\x03\x00\x00\x00\x00\x00 (block's hash string converted to bytes in little endian order), I am getting the following content which I hexlified for readability:

87c628858c201d04009be99e5883d1e92c0100000050120119172a610421a6c3011dd330d9df07b63616c2cc1f1cd00200000000006657a9252aacd5c0b2940996ecff952228c3067cc38d4885efb5a4ac4247e9f337221b4d4c86041b0f2b5710

The first 3 bytes (87c628) seem to be the same for every block that I get from the database. I'm guessing it is some version identifier?

The last 80 bytes (or 160 characters in hex) are the block header: 010000005 ... f2b5710. They match what you get by executing bitcoin-cli getblock 000000000003ba27aa200b1cecaad478d2b00432346c3f1f3986da1afd33e506 false.

The rest is a mystery to me. I know that block 100,000 had 4 transactions, but the byte that holds the value 4 has 129 for block #1, which had only a single transaction.

Where can I find out about the format of, at least, the block keys in LevelDB?

Could it be that the format of each key depends on the version of bitcoind that serialized the contents? If so, how do I figure out which format to use?

Arjen

Posted 2018-01-04T21:29:42.757

Reputation: 143

LevelDB has its own database format and compression stuff that may make reading the data directly from disk as binary data to be difficult. You're probably running into some of that stuff. – Andrew Chow – 2018-01-04T22:42:11.217

You're right, that's why I am reading the LevelDB using a Python library (plyvel). That seems to be working correctly because I can retrieve records for blocks (bX*32 records) and files (fXXXX). I want to use the info stored in these records to seek through the .blk files directly. – Arjen – 2018-01-05T08:55:20.933

Arjen can you please publish your code for retrieving the records using plyvel? – Adam Rosenthal – 2018-08-18T18:59:41.867

Answers

4

This is the Bitcoin Core v0.15.1 source code snippet that defines the serialization of CDiskBlockIndex. The various fields are:

  • A varint version number of the client who wrote the record (for future extensibility). This is the 87c628 you're seeing.
  • A varint block height.
  • A varint status with various flags.
  • A varint determining the number of transactions in the block.
  • If the status field has bits 8 or 16 set, a varint determining which file (blk*.dat and rev*.dat) the block is stored in.
  • If the status field has bit 8 set, a varint determining the offset into the blk*.dat file where the start of the block is.
  • if the status field has bit 16 set, a varint determining the offset into the rev*.dat file where the start of the block's undo data is.
  • The block header

Pieter Wuille

Posted 2018-01-04T21:29:42.757

Reputation: 54 032

I've now managed to decode the first five fields: version, height, status, num of txs and block number. I'm getting incorrect values for the data position, but I now know that I'm on the right track. Thank you for taking the time to answer. – Arjen – 2018-01-06T14:06:58.913

I was a bit too fast: I had a bug in my VarInt parser. All is good now. Thanks again! – Arjen – 2018-01-06T14:39:07.543

Asked: 2018-01-04T21:29:42.757

Viewed: 674 times

Active: 2018-08-18T18:59:41.867