1
I am confused on what incentive a miner has on including a transaction apart from the transaction fee. So, if adding a transaction wastes any time, any miner should ideally not add transactions that give no or very less mining fees since it is only wasting time.
Am I missing some subtle point in how this system works?
2
A miner checks whether all of the inputs of an incoming transaction exist in the UTXO set. If at least one doesn't, the transaction is necessarily invalid. This doesn't necessarily mean the transaction has already been included in the blockchain, it can also mean that that UTXO never existed or that it has been spend in a transaction which isn't the same as the incoming one.
A miner doesn't even have to traverse the UTXO set because there are smart things like hash maps.
A miner's main incentive to include transactions is the transaction fee. Of course, keeping Bitcoin going would also be nice but if there's money in the game, most people go for the money. This is why miners prefer to include transactions with a higher fee and why there are recommended fees for getting a transaction confirmed within a certain time frame. If you want your transaction to be included right in the next block, you should pay a fee high enough so that there are less than [1 MB - your transaction's size] of other transactions which are more profitable (more fee per storage space) than yours because 1 MB is the block size limit. If you got more time, you can choose a smaller fee because random fluctuations in the time which passes between blocks will cause there to sometimes be more and sometimes to be less competition for transactions to be included in a block and yours can make it into one when there is less competition a few hours in the future for a smaller fee.
0
A miner doesn't have to check whether a specific transaction has been included already. He just has to check whether the transaction is valid.
A transaction for which any inputs have already been spent is not valid. If a transaction was included previously, its inputs are already spent, thus it's not available for inclusion again.
