You're right, your wallet is only holding your private keys and tracking the unspent transaction outputs that you're able to sign for. The UTXOs are also tracked by every full node on the network, which have digested the complete transactional history of the network to learn exactly which pieces of the currency are currently available for spending. As long as you maintain control of the private keys, you can regenerate the information of the spendable balance from the publicly available ledger.