In your scenario there are two transactions with the same input, i.e. an attempt to double-spend.
If the first transaction is already part of a block in the chain, the second one can not be included in any block on top of that block. Other nodes would reject that new block. A miner could build his new block on top of the last block before the first transaction, but that fork of the blockchain would never catch up with the original chain, unless more than 50% of the networks hashrate also builds on top of it. That's the mechanism preventing double spending.
If the first transaction is not yet part of the blockchain (i.e. 0 confirmations), a miner can choose which one to include in the next block. Choosing the one with the higher fee would be a sensible choice, but I don't know if that is implemented in any client. In that case the first transaction would be discarded, and never become part of the blockchain, since it's effectively the same scenario as above.
So, no, there is no potential for a double spend attack through higher transaction fee.
AFAIK They would choose the one they received first, the oldest transaction. – Blazr – 2012-05-10T12:41:52.130
Would doing as you describe make other honest miners reject the block, assuming they received the other transaction first too? – Highly Irregular – 2012-05-10T23:07:41.990
1That would be a quick way for a pool to commit suicide, I suppose. The bad PR as a result would probably be fatal. – Stephen Gornick – 2012-05-10T23:26:13.400
3@HighlyIrregular No, as the transaction would still be valid. A pool can put any valid transaction in the block, even if it invalidates some other transactions which are not part of any block. Otherwise there would be a fork each time someone would perform a double spend attack. – ThePiachu – 2012-05-10T23:48:48.333
2@StephenGornick It certainly would be, however, if someone make a private pool for themselves that would prefer higher-paying transactions, they could still do it. Probably some miners would still join it in hopes of getting slightly higher payouts. – ThePiachu – 2012-05-10T23:50:14.243
1Then I guess that's a good reason to wait for extra confirmations if you're receiving a substantial sum of bitcoins from an untrusted party. – Highly Irregular – 2012-05-11T03:41:33.380