Creating my own bootstrap.dat?

Since I have been tearing down and setting up bitcoin clients a lot for some experiments lately, I was wondering whether there is a simple way to speed up the synchronization with the network. One method that comes to mind is the bootstrap.dat file distributed by some people. The problem is that the last 40'000 blocks, which are the most expensive to get and verify, are not included.

Now I was wondering how to create such a bootstrap.dat myself, from a good copy I have on another machine. Any idea?

cdecker

Posted 2013-04-29T17:37:44.117

Reputation: 7 878

Answers

The block files are in the same format as the bootstrap file. Their format is really simple: just concatenate all blocks after prefixing them by the network number (to avoid mixing the testnets) and the block length.

[network number] [length] [block header] [block transactions]
[network number] [length] [block header] [block transactions]
[network number] [length] [block header] [block transactions]
... repeat for all blocks

To create a bootstrap file, all you need to do is to just grab the data from blk00001.dat up to your last block file (here that's blk000054.dat) and put all of it in the bootstrap.dat file, preferably sorted so that it can just import them all in one go without having to swap blocks.

I think you might even be able to just concatenate the block files themselves, as they are already in the right format, and the format supports concatenation.

Tom van der Woerdt

Posted 2013-04-29T17:37:44.117

Reputation: 2 397

Didn't know of this functionality, makes me wonder what's the point of these files since you can always download an archive with all blkXXX.dat files and uncompress it at the appropriate directory, getting exactly the same results except without the renaming to ".old" at the end :-S – Joe Pineda – 2014-03-05T11:35:56.453

Bitcoin Core contains a tool to do this properly (filtering out any orphan blocks, putting everything in the correct order — which will work a lot better than just concatenating the block files. It's documented in its own README, here: https://github.com/bitcoin/bitcoin/tree/master/contrib/linearize

In short:

Launch Bitcoin with an rpcuser and rpcpassword set in your config file, with the correct location as per https://en.bitcoin.it/wiki/Running_Bitcoin#Bitcoin.conf_Configuration_File
Checkout the Bitcoin source from Github and cd to contrib/linearize.
Copy example-linearize.cfg to linearize.cfg and update it with your rpcuser/rpcpassword values, along with the maximum block height you want to output (max_height), and the path to your data directory (input), as well as the path to your desired output file (output_file).
Run ./linearize-hashes.py linearize.cfg > hashlist.txt followed by ./linearize-data.py linearize.cfg.

Peter Josling

Posted 2013-04-29T17:37:44.117

Reputation: 201

3After spending a HUGE amount of time on this issue, Peter Josling's answer needs upvoting more. Concatenating the files from a node without sanitizing them leads to unexpected results, such as the syncing client jumping to network download after an orphan branch is detected. My rep will not allow me to comment or upvote, but anyone building a bootstrap file really needs to be aware of the pitfalls of simply concatenating the blk*.dat files – lavajumper – 2017-07-01T20:42:18.520

in 2019 this should be the right answer. Thank you – Al Po – 2019-01-05T12:52:43.590

 From Mac

Open terminal
Go to the folder where the blocks are. In this case Bitcoin's folder (it's the same for other bitcoin forks)
```
cd "Library/Application Support/Bitcoin/blocks/"
```
Run this code that will concatenate all the blk files. Make sure to include all the numbered blocks you find on your Bitcoin's folder. In this case there are 4
```
cat blk000*.dat > bootstrap.dat
```
Done.

David Lopez

Posted 2013-04-29T17:37:44.117

Reputation: 159

Making your own bootstrap.dat is fairly simple. In windows, drop to your command prompt and use the following command.

Copy /b blk00000.dat+blk00001.dat+blk00002.dat bootstrap.dat

You'll want to keep going with the + blk*.dat files until you reach the last one. To make things easier for you, just run a batch file which can take care of adding them all without having to input each blk*.dat file. Simply copy and paste the code below into a text file and name it something like Create_Bootstrap.cmd and run it from your /blocks folder.

@echo off 
setlocal enableDelayedExpansion 

set BITCOINDIR=%APPDATA%\bitcoin\blocks
for /F %%x in ('dir /B/D/ON %BITCOINDIR%\blk*.*') do (
  IF NOT [!B!] == [] set B=!B!+
  set FILENAME=%BITCOINDIR%\%%x
  set B=!B!"!FILENAME!"
)
copy /b %B% bootstrap.dat

or just use "copy /b blk*.dat bootstrap.dat"

m0gliE

Posted 2013-04-29T17:37:44.117

Reputation: 29

Go to en.blockchaindownload.nl and download the newest chaindownload. On the site is also a guide that tells uou how it works.

Just put that file onto a USB and you can use it over and over. Its free to use and updates the download twice a week. It uses torrents with 2 or 3 (depending on the torrent you pick) webseeds for all the speed you need.

rollo1996

Posted 2013-04-29T17:37:44.117

Reputation: 127

-1

It is somewhat intuitive what files need to be copy and where once you have a full node built. What is not so intuitive is how to bootstrap a full node before you have built your first full node. You first need to know where the files need to be placed for the various 'OSes'. I'm not sure if there are little or big endian issues with copying the raw Blockchain node files between different OSes.

Another approach is to let the protocol handle the problem. Secondary and tertiary nodes build very quickly if they a on your own high bandwidth LAN. Just point a secondary node (by tweaking its bitcoind.conf file) to the primary node that is already built using the "connect=Your_IPv4_LAN_Address". Such builds move very quickly... You might find it to be faster than writing the ledger to a thumb drive and copying it to another computer.

skaht

Posted 2013-04-29T17:37:44.117

Reputation: 2 588