2
There are two answers to your question.
For a signed message, you can recover the public key if you have both the signature and the message that was used to create the signature. The algorithm actually returns four possible keys, so the signature includes a header byte to identify which of the four should be used. This is possible because we know which curve (secp256k1) is used by Bitcoin when it generates the signature. If you want to see how this is done, look at recoverFromSignature() here
For a transaction, the public key is provided in the input script, the output script or a hashed script. In the case of a multi-sig transaction, there are multiple signatures and multiple public keys. There are also multiple ways to sign a transaction as determined by the signature hash type (a control byte added to the end of the signature). If you want to see how this is done, look at checkSig() here
The signed message is the original message that was used to generate the signature. I'll change the answer to specify the original message. – ScripterRon – 2014-03-29T01:28:01.040
You are talking about message signatures. The uestion is about transaction signatures, which do not use key recovery and do not have a header byte to indicate which public key is to be recovered. – Pieter Wuille – 2014-03-29T13:54:03.283
2@Gracchus: Bitcoin (the reference client, and some other software) has the functionality of cryptographically signing messages using Bitcoin keys (with associated Bitcoin address). This has nothing to do with transactions or the peer to peer network though, it's purely a local operation. For these message signatures, a more advanced signing technique is used that wasn't known when Bitcoin transactions were designed. – Pieter Wuille – 2014-03-30T22:19:37.373
From a malleability perspective, I think this is a moot point: if a valid public key and signature is not provided, then the transaction can't be valid anyway (finding multiple valid public keys for an address is near-impossible). Still, better documentation on what is signed when/where/by whom would be great. – Tim S. – 2014-03-28T16:57:56.487
Are you asking about signing transactions or signing messages? – ScripterRon – 2014-03-29T15:32:54.597
@Gracchus As Peter noted, the answer is incorrect if you are asking about transactions. Bitcoin does things differently when signing transactions versus signing messages. The public key is provided separately (either in the input script, the output script or a hashed script) – ScripterRon – 2014-03-30T12:15:38.753