Any node (not just a miner) can check whether a transaction's inputs have been previously spent by looking through the entire blockchain to see if any transaction has spent that input. There is no correct or incorrect way to do this; not all nodes have to do this in the same way.

The common way to do this is to trust yourself (as a node) to have verified the blockchain correctly in the past. What Bitcoin Core (and other software that follow Bitcoin Core's methods) does is it maintains a separate database that contains all of the unspent transaction outputs. So if the previous outputs that a transaction's inputs references are not in this database, then the node declares the transaction invalid. It assumes that its database is correct because it built that database.

There is nothing that forces miners to do anything. A miner could create an invalid block that contains an invalid transaction (e.g. one that spends from a non-existent output). However all nodes (not just miners) verify that blocks and transactions are valid. So the Bitcoin network will reject any invalid blocks and thus the miner will not actually get his reward for his work. This incentivizes miners to verify block and transactions, but they aren't forced to.