Can Bitcoin blocks be organized into shards?

Bitcoin Scalability is a hot topic. In database architecture design data is segmented into shards or the data is partitioned such that it can be spread among large number of machines.

In the financial world, credit card companies like Visa have to deal with thousands of transactions per second. To deal with this volume credit card processors will use the Date as a shard key. This allows for the credit card processor to group all transactions made in a set period of time and a set of machines are made responsible for a month of transactions. Is this sharding tactic compatible with Bitcoin blocks? Or more generally what if Bitcoin was as popular as credit cards? What kind of techniques would Bitcoin be forced to use to cope with the scale of the problem?

Rook

Posted 2012-06-19T04:01:28.093

Reputation: 261

I gave it some more thought - this is the result: https://bitcointalk.org/index.php?topic=88714.0

– ripper234 – 2012-06-20T14:31:05.620

Answers

If you mean whether block verification can be distributed so that one entity can participate without storing the entire blockchain, this isn't compatible with the current design, but maybe a new design can do that. Also, suggestions such as this aim to improve the ability of lightweight (SPV) clients, who do not store the entire blockchain, to verify transactions concerning them specifically, without changing the core protocol.

If you mean whether a single entity can distribute the blockchain among several machines on the same LAN, then this is possible. The basic operation needed for block verification is to look up the details of a previous output, so the natural sharding key would be output hashes. One could also cache the newest outputs which are the most likely to be used in the following transactions.

Meni Rosenfeld

Posted 2012-06-19T04:01:28.093

Reputation: 18 542

Now that I think about it, perhaps N different (but very similar) blockchains could be created. If an easy API existed to trade between the chains in a secure, atomic way, then you could send money between any two chains by trading to the target chain first and then sending a normal tx. – ripper234 – 2012-06-20T14:09:30.780

https://bitcointalk.org/index.php?topic=88714.msg977222#msg977222 – ripper234 – 2012-06-20T15:54:17.377

There is nothing so special about the data in the block chain that this cannot be done, but at present I know of no client which implements such a thing. BitcoinSharp, BitcoinJ and the default (Satoshi) client all use different database types and there's no reason you couldn't make a client that used still another database type.

So at present, no, it's not possible with off-the-shelf software, but there's no reason it couldn't be implemented as such if scale became an issue, or even implemented today as a proof-of-concept.

David Perry

Posted 2012-06-19T04:01:28.093

Reputation: 13 848

OP might have meant changing the Bitcoin protocol to allow each miner to process a particular shard, easing/solving its scalability requirements. See my answer. – ripper234 – 2012-06-19T06:27:31.120

The blockchain system IS a distributed database with its own architecture. – Rook – 2012-06-19T08:09:15.550

I don't think it will work.

I don't understand what you mean by sharding by date. The bitcoin analogy would be sharding by block-number ... but that is already the case ... every block is worked on separately. The thing is, every block depends on the previous block, so a miner has to keep some record of the entire blockchain history.

I initially thought you meant just sharding by "shard ID" - splitting the blockchain into N independent chains, where a miner can decide to work on "subchain 137 out of 1000". This idea won't work because every Bitcoin address must be able to send money to every other Bitcoin address. So, in order to mine only one shard, you still have to know the state of every other shard, because there will be transactions linking these shards (there could also be tx linking multiple addresses split between multiple (>2) shards).

Lastly, if you aren't suggesting a fork in this question, but just asking about techniques to process the existing chain structure using shards ... yes, the blockchain's data is shardable, it could be sharded by address. A miner would still have to process each new block, but its data set could be stored on a Redis cluster instead of in the memory of a single machine.

Update - then again, maybe I was wrong. Here are my latest thoughts on the matter.

ripper234

Posted 2012-06-19T04:01:28.093

Reputation: 25 192

maybe you should read about sharding before posting. – Rook – 2012-06-19T08:10:05.987

@Rook - thanks for that ever helpful comment. – ripper234 – 2012-06-19T11:39:15.130

It would be possible in a few areas:

Storing the block chain on a server could be done in the same manner any data is stored - be it in one places or in many. This would be trivial from Bitcoin perspective.

Transmitting the data could also be partitioned into shards should a small change to the protocol be made - allowing one to request a block header with the information about the full merkle tree. This way one would know exactly which transactions are stored in the block without the need to know everything about all the transactions. The rest of the data could be fetched as needed from other places. This is similar to pruning the block history.

Preparing the blocks for mining could also be handled in a similar way, but it would require more trust from other nodes. A pool would only request the transaction hashes from other clients, not really needing to see the entire transaction in order to include it in the block header + merkle tree structure. The only downfall of this would be that someone could sneak in a few invalid transactions in this way, as there would be no data checking. So all in all, the pools that would wish to operate on seeing only part of the data would probably have to limit themselves to including transactions they have seen. This would mean that some transactions might take longer to include than others, depending on which pool has seen them.

All in all, Bitcoin's data could be partitioned with only small changes to the protocol, but that would require a lot of consideration first.

ThePiachu

Posted 2012-06-19T04:01:28.093

Reputation: 41 594