Bitcoin uses its own wire protocol, which is based on TCP. It's quite simple, with each packet type having strict rules for conformity and a checksum. New objects (transactions and blocks) are flooded through the network to all peers with a flag indicating they want that information, along with auxiliary messages like peer addresses and pings. No encryption is used for any connection.

https://bitcoin.org/en/developer-reference#p2p-network

It's a fairly simple custom protocol. You can find details at https://en.bitcoin.it/wiki/Protocol_documentation, but the messages use this general form:

• magic, 4 bytes, uint32_t: Magic value indicating message origin network, and used to seek to next message when stream state is unknown
• command, 12 bytes, char[12]: ASCII string identifying the packet content, NULL padded (non-NULL padding results in packet rejected)
• length, 4 bytes, uint32_t: Length of payload in number of bytes
• checksum, 4 bytes, uint32_t: First 4 bytes of sha256(sha256(payload))
• payload, variable length, uchar[]: The actual data

The "inv" command is used to send notifications of new transactions and blocks around the network, and the "getdata" command is used to retrieve information about those transactions and blocks.