The lifecycle of a transaction is the following:

1. Transaction creation:
   A sender creates a transaction, he includes a list of recipients, determines which previous transaction outputs to spend, and signs it. The difference between output total and input total determines the transaction fee.
2. Relay:
   The sender submits the transaction to the Bitcoin network. He simply sends it to his peers, which in turn relay it to theirs. Eventually the transaction will be seen by everyone as it spreads through the network.
3. Transaction selection:
   From all transactions that a miner is currently aware of (the "mempool"), each miner selects a subset that they want to include in a block. They usually select to include as many as are allowed (1MB), and select to create the most revenue for themselves (highest fee first). The miners are allowed to make any selection they want though, even to include no transactions at all (except the required coinbase transaction).
4. Block discovery:
   Approximately every ten minutes, one of the miners succeeds at authoring a block. This block confirms the transactions that he previously selected by writing them to the blockchain (as long as this block ends up being part of the chain with the most work). This block in turn is relayed through the network to update all nodes of the new state of the network. The payment is considered settled at this point (with a high degree of confidence, although a few extra confirmations don't hurt).

There are multiple ways for a transaction to fail getting confirmed along the way:

1. The transaction is invalid, e.g. by spending unavailable money, being malformatted, being in conflict with a confirmed transaction, or the signature being incorrect. Nodes will not relay invalid transactions and if they already have them, drop them from their mempool.
2. The transaction doesn't get relayed. This happens e.g. when the transaction fee is below the minRelayTxFee of the sender's peers. The senders will not add such a transaction to their mempool and will not relay it.
3. The transaction doesn't get selected for block templates. If a transaction doesn't belong to the top 1MB by transaction fee rate, it will usually be delayed significantly, until the feerate comes down to its own, it is made invalid by a competing transaction, or a miner is incentivized in another way to include it in a block. Such transactions may time out eventually (How do transactions leave the memory pool?), but can remain valid indefinitely.

So, 1) there is a strict limit in the supply of blockspace with 1MB per ~ten minutes, and 2) Bitcoin transactions are useful in a number ways, it stands to reason that there will always be some sort of minimum bid to use blockspace (e.g. consolidation of low value unspent outputs, timestamping, colored coin applications, decentralized backups, poetry…). It follows that there will always be more demand for blockspace than supply and the price for blockspace will find some sort of balance at a fee rate where demand and supply match. Therefore, there will always be some unconfirmed transaction.