Does bitcoind send message headers and payload separately in P2P protocol?

0

I am connecting to my local bitcoind using TCP and observed the following behavior:

I see many messages where only header is given or only part of the payload is attached. Of course, it could be a bug my code too.

Are the bytes sent in chunks? Which of the following are valid?

|header1|payload1|     (as per documentation, this is valid)
|header1|              (no payload, even though required. Have seen this)
|payload1|             (no header, have seen this)
|header1|partPayload1| (incomplete payload, not sure if seen these)
|partPayload1|         (part earlier header's payload, not sure if seen these)
|header1|payload1|header2|payload2| 
|header1|payload1|header2|partPayload2|  
|header1|partPayload1|header2|
|partPayload1|header2|payload2|

Note: I am using version number 70002, relay = 1 and services = 0.

EDIT: Generally I wait for the remaining bytes, and most of the times they arrive, but in rare cases, around 2%, some other header arrives. This happens only with tx messages.

EDIT2: Based on the comments below, it seems likely that there is a bug in my code. I'll check and revert.

EDIT3: Was a bug in my code. I was assuming that partial data packets correspond to a single header. There can be multiple headers after the data is complete (in the same packet). This makes sense once I consider it as a stream.

Jus12

Posted 2018-01-09T11:08:51.923

Reputation: 1 243

Answers

1

TCP is a stream protocol. Even though on the wire the stream is submitted as a series of IP messages, semantically it is just a stream of bytes.

This means that at the application level you shouldn't care about the message boundaries. Routers and other internet infrastructure can arbitrarily split up data into packets to fit in their underlying protocols (e.g. Ethernet has frames of at most 1500 bytes).

Pieter Wuille

Posted 2018-01-09T11:08:51.923

Reputation: 54 032

I am reading the data from socket and it comes incomplete (the first three cases, I have observed when connecting using TCP). – Jus12 – 2018-01-09T11:19:16.873

What do you mean by incomplete? It's just a stream of bytes. If a read from the socket returns and you don't have a full Bitcoin P2P message yet, it means the rest will come later. – Pieter Wuille – 2018-01-09T11:21:06.570

The header checksum verifies, says payload should be x (non zero) bytes but the message is either only 24 bytes or less than 24+x. So I need to wait for the rest of the message, with the guarantee that the peer has actually send the complete data and its on the way, and it won't send header of a different message. Right? – Jus12 – 2018-01-09T11:21:53.120

Then that means you haven't seen the entire message yet. You'll see the rest in a subsequent socket read. – Pieter Wuille – 2018-01-09T11:22:26.803

Edited my question to explain the symptoms instead of extending the comments. – Jus12 – 2018-01-09T11:27:04.300

What do you mean by "another header arrives"? If the previous message isn't complete yet, then the first bytes that follow belong to that message. – Pieter Wuille – 2018-01-09T11:29:11.573

The next message instead has the magic bytes, while the earlier payload was not fully received. Is it possible that the packet was lost? – Jus12 – 2018-01-09T11:31:06.670

That sounds wrong. Bitcoind should not do that (there can be a bug of course, but that seems unlikely as it would be very noticable). – Pieter Wuille – 2018-01-09T11:32:32.677

Does bitcoind send messages as follows? |header1|payload1|header2|payload2| or is it always only one message at a time? – Jus12 – 2018-01-09T11:55:12.533

Why does it matter? TCP is a stream of bytes. You can't tell in which way those bytes were broken up in syscalls by the sender, and you shouldn't care. – Pieter Wuille – 2018-01-09T14:28:39.860

Asked: 2018-01-09T11:08:51.923

Viewed: 103 times

Active: 2018-01-09T19:41:22.523