When we can agree on mempool sorting order, we can just confirm first n-transactions that are at least 10 mins old. What's missing?

When we can agree on a mempool sorting order, we can just confirm first n-transactions (or 80% of transactions) that are at least 10 minutes old. Then, we can transfer (broadcast) the block header and a hash to the set of all confirmed transactions.

Just consider this as an aggressive version of BIP152, compact blocks.

This kind of method will work for transactions of any size; both confirming 10 transactions and 1 million transactions are the same.

What's missing? Why are we not doing 'confirmation' like this?

For example, if we have 1000 transactions in a mempool (sorted), 900 transactions are at least 10 minutes old, and we're confirming the first 720 transactions.

This may not be a perfect or complete solution, but we just need to find,

Sort order; fee, size, etc.
Transaction age; say, 10 min
Confirm size; say, confirm first 80%

Note: Sort order is based on f(trxn-meta-data; fee, input, output, etc.), transaction age is used as a filter. Another filter is first x% of the set.

vi.su.

Posted 2016-07-07T08:41:54.723

Reputation: 1 714

1its up to the node to sort it in anyway he likes. Perhaps a sorting based on fee would make more sense to them. – abeikverdi – 2016-07-07T08:53:10.627

@abeikverdi I'm asking for sorting order consensus, and get rid of block size debates forever. – vi.su. – 2016-07-07T09:14:36.907

1What advantage do I get if everyones mempool is sorted based on a certain criteria? What does it have to do with block size limit? – abeikverdi – 2016-07-07T09:34:09.643

1By sorted you mean two nodes have the same transactions in mempool or you mean transactions are listed and sorted based on a factor? – abeikverdi – 2016-07-07T09:35:24.120

When all mempools are sorted, most likely all mempools are in sync with unconfirmed first n-transactions. This fact can be then be used to announce these transactions are confirmed, along with block header and a hash to all these newly confirmed transactions. – vi.su. – 2016-07-07T09:37:51.860

Idea is to get first n-transactions in every mempool the same, so that we can auto confirm first 1k, or 1M transactions without building and broadcasting blocks. – vi.su. – 2016-07-07T09:41:39.370

You made me confused with "sorted". First of all there is no may to make sure that everyone has received the broadcasted transaction, therefore always there are some transactions missing for each node. IBLT could help them to sync the mempool and stuff. But in general what your saying is basically what a blockchain is trying to do. We create the blockchain and all the proof of work and consensus to agree on the transactions that cant simply be synced. – abeikverdi – 2016-07-07T09:52:46.457

What happens if there are double spent in the list of first n transactions in mempool? – abeikverdi – 2016-07-07T09:53:44.900

it is not possible to have 'sorting order consensus' or 'mempool synced' because of https://en.wikipedia.org/wiki/CAP_theorem

– amaclin – 2016-07-07T12:16:26.283

@amaclin interesting. I'll look into it. :) – vi.su. – 2016-07-07T14:09:32.327

Answers

Your answer assumes different nodes can have a consistent view of the mempool. If that were the case, we wouldn't need a blockchain at all, whose sole purpose is establishing consistency between different nodes' view of history.

The reason this is not possible is due to the laws of physics. A transaction tx1 broadcast in Australia, which conflicts with a simultaneously-created transaction tx2 broadcast in Alaska cannot be both included in a block. However, nodes in the rest of Asia will see tx1 first, and nodes in the rest of North America will see tx2 first. Which of those two is legitimate?

There is no solution to that problem, as due to general relativity, the concept of 'first' depends on one's frame of reference. A proof-of-work controlled blockchain solves this problem by accepting that not all nodes need to immediately agree on which of those transactions is accepted, and instead make economics force miners to make a choice between the to candidates.

Despite not having a guarantee for consistency between mempools, there is of course a very strong overlap. This overlap is exploited in many of the proposed efficient relay protocols, including the Relay Network, IBLT, Block network coding (and Compact Blocks which you reference).

Pieter Wuille

Posted 2016-07-07T08:41:54.723

Reputation: 54 032

Thanks for the answer. May be, my question is not clear. We still need blockchain to distribute rewards, and every 10 mins we still need to broadcast blocks (header only). – vi.su. – 2016-07-07T11:37:08.820

I understand, everyone's frame of reference will be different, but my assumption is, a sorted set, that's at least 10 mins old should be the same across all miners. – vi.su. – 2016-07-07T11:40:01.063

2If you have a consistently synchronized mempool across the globe, it means you can use it as a single shared synchronized database. You don't need blocks anymore to distribute updates - just use the mempool as the state of everyone's account directly. – Pieter Wuille – 2016-07-07T11:40:26.213

1@vi.su If the top 80% sorted mempool you refer to is not guaranteed to be exactly the same across the world, your idea does not work (and physics dictates that this can't be the case). What does work is protocols that find the (small) differences between mempools, and transfer the differences (which all of the linked proposals in some form or another do). – Pieter Wuille – 2016-07-07T11:41:48.610

transactions can be sorted without time, if they could include previous transaction (any tx) from mempool. Just as a proof of, our transaction was created after a specific unconfirmed transaction. Just like, previous block in blocks header.. – vi.su. – 2016-07-10T09:22:45.780

There are a few issues with this proposal:

No transaction could ever be confirmed in less than ten minutes.
We already have plenty people up in arms that choose to rely on zero confirmation transactions. Waiting at least ten minutes would increase confirmation times.
Every node has their own mempool and they don't necessarily match.
Every node receives transactions at different times and perhaps in different orders, so some transactions might not have come to the attention of all nodes yet. Especially, the age of transactions would not be the same in all mempools. There also might be some doublespends floating around, which could make it impossible for two nodes to match their mempools.
A single hash is insufficient to reconstruct the confirmed transaction set.
As soon as there would be any difference in the mempools the recipient node would fail to reconstruct the block. Besides, if a single hash were sufficient the block header already includes the Merkle root of the transaction tree.

These related ideas come to mind:

Gavin Andresen proposed an Invertible Bloom Lookup Table based set reconciliated a few years ago. It was based on a fixed transaction order in blocks.
The weak block proposal has miners also broadcast unsuccessful blocks to announce what transactions they are working on.

Murch

Posted 2016-07-07T08:41:54.723

Reputation: 41 609

Thanks, I've heard about IBLT, but never understood it properly. I'll look into it. – vi.su. – 2016-07-07T09:26:48.613

10 min is arbitrary, it's chosen as an example to have matching block reward cycle. When someone can process 0-conf with blocks, then they can also process with conf-set from mempool. – vi.su. – 2016-07-07T15:53:37.827

Still, I couldn't understand why from a sorted set, first 80%, transactions that are at least 10 mins old goes async. – vi.su. – 2016-07-07T15:57:43.710

Single hash (not Merkel root) is used here just to verify the set, not to reconstruct, as the transactions accross all mempools are the same.. – vi.su. – 2016-07-07T16:00:23.467

1Let's say, you heard about transaction X at 14:10, I got it at 14:11. When you find a block at 14:20 you include transaction X, but for me it's not available for inclusion yet. Transactions don't have a timestamp, so there is no universal information how old they are. – If you just want to verify the set, you can do that with the Merkel root as well, both the Merkle root and a single hash over all transactions are a digest of the complete transaction body. Neither is sufficient for reconstruction, both are for verification. – Murch – 2016-07-08T08:36:42.187

We're not sorting mempool based on transaction received time, so our time when we receive the transaction doesn't make any difference, unless the difference is >10 mins. Sorting order may be, a combination of utxo sort, fee, bitcoin days destroyed, oldest first, from address, to address, trxn amount, etc. – vi.su. – 2016-07-08T10:07:43.643

1Actually, you can think of the 10 minute age as a second dimension of sorting: You sort into the two buckets "older than 10 minutes" and "younger than 10 minutes". As I said before, there is no other time information attached to transactions than the time our node first saw it. Therefore, every node will sort differently in the time dimension, even if they all sort the same way in your proposed mempool order. It follows that you cannot guarantee a matching set of transactions ready for inclusion. – Murch – 2016-07-08T12:47:58.573

no two buckets. I mean, sort irrespective of time and then filter it by top 80%, then remove new transactions. This is just an example, but we need a method that would not conflict accross miners. – vi.su. – 2016-07-10T09:38:23.120

1"remove new transactions" as explained multiple times above, this can't be done, because "we need a method that would not conflict across miners", but blocks and the blockchain are what resolves the conflict. – Murch – 2016-07-10T22:01:09.747

1understand now. Remove is not possible. Some will not remove 10 mins old trxn, that could be 9 mins old in my mempool. – vi.su. – 2016-07-11T05:51:28.047