The other answers describe the rules currently implemented in the client, but I expect those rules to be replaced once block size limits become relevant. Since those rules aren't enforced when clients verify the blockchain, they are only guidelines, and will be violated if they're against the interest of miners.
Miners will simply choose the subset of candidate transactions that respects the limits (blocksize and signature count) and maximizes the fee.
Disregarding a few minor complications (knapsack and dependent transactions) this means that miners sort the transactions by fee/transactionSize if the maximal size is the limiting factor, and fee/signatureCount if the signature count is the limiting factor.
That rule is pretty strange, and doesn't look stable to me, since it's against the miners interests. – CodesInChaos – 2012-04-09T20:24:25.007
3It's not against the miner's interests. If the miner fills up the block with low-fee transactions, he loses the opportunity to get high-fee transaction in there. Remember, not all transactions are the same size, so you can't just go in fee order. (The optimum algorithm would be to sort the transactions into two pools, one for fee transactions in fee-per-byte order, the other for free transactions in priority order. Fill a block from the first pool in order if possible until no more paid transactions fit, then take as many from the second pool as fit.) – David Schwartz – 2012-04-12T05:23:07.777
@DavidSchwartz I argue that miners will converge to a fee-per-byte auction instead of this algorithm because that's in their interest. You agree with the fee-per-byte part, but disagree with the current algorithm being against miners interest. So your comment seems inconsistent to me. – CodesInChaos – 2012-04-16T15:18:17.750
@CodeInChaos: The effort required to act in their interest would actually make it not in their interest overall. The difference between an ideal algorithm and the current one, today, is about 0.02% or so. It's like driving across town to pay a penny a gallon less for gas. – David Schwartz – 2012-04-16T18:20:12.937
Even if the algorithm wouldn't require a fee, if the the coin used for input is new a fee still could be required: http://bitcointalk.org/index.php?topic=84434.msg931988#msg931988
– Stephen Gornick – 2012-05-31T03:56:52.920