The rule is that the longest* valid chain is considered the active one. If you build a forking branch with invalid transactions in it, the network (in particular, all full nodes - including miners) will simply ignore it. If it's invalid, it does not exist. Blocks are not relayed to peers before validating them entirely.
You may be able to temporarily fool lightweight clients that don't do full validation, but to build a chain with sufficient length to "confirm" these invalid transactions requires sustained high hashrate, assumed to be impossible without significant (close to or above 50%) mining power.
longest* = it's actually the branch with the highest estimated total number of hashes that is considered the active one (as a tie breaker), not the longest one. This only very occasionally makes a difference (in particular, when the fork crosses a retarget).
Ok thanks for your answer. When you say "the total number of hashes", you mean in the last found block or in the whole blockchain? – heyo – 2014-07-22T23:08:57.710
In the whole blockchain. Fun fact: that number recently surpassed 1 yottahash (a 1 with 24 zeroes). – Pieter Wuille – 2014-07-22T23:11:14.283
another question: when are the transactions checked? When the block is found, when the emitter of the transaction sends it to other nodes or the both? – heyo – 2014-07-22T23:11:55.070
Every node validates it before sending it to other nodes. – Pieter Wuille – 2014-07-22T23:12:37.773
Ok but if I create a custom-fake block with fake transactions, successfully mine it and say to the network I have the answer, they're gonna compare the merkle root, no? My question is : how are they gonna find out if it's a fake and ignore it. – heyo – 2014-07-22T23:18:13.153
Because they validate it. – Pieter Wuille – 2014-07-22T23:53:16.423
Longest above sufficient difficulty! ;)) +1 – None – 2014-07-23T17:49:17.397
@heyo When you say "fake transactions", what do you mean? In what sense are they fake? How are they different from valid transactions? – David Schwartz – 2014-07-23T17:59:36.810
Fake because they're present twice (double spending). If I understand well, miners can validate blockchain in a fraction of second, Proof Of Work is only there to slow down and regulate the distribution of Bitcoins that need to be created. – heyo – 2014-08-14T17:44:57.127
1Adding the same transaction again in a block within a block chain that already has that transaction is a classic double spend attempt, and simply makes the block invalid. No node or wallet in the network will accept it. – Pieter Wuille – 2014-08-14T17:50:45.027