A Bitcoin private key always have exactly one corresponding public key from which the Bitcoin address can be derived. The derivation paths looks as this:
Private key -> public key -> Bitcoin address. If you e.g. only have the public key you can determine the bitcoin address, but not the private key. Given only the Bitcoin address you can in fact not even determine the underlying public key.
Therefore all you actually need to save is the private key, from which you can find the public key/address and check if there are any unspent outputs for it.
On top of that deterministic Bitcoin wallets have a clever way to derive determanisticly a nearly infinite amount of private keys, so all you then would need to keep safe is the seed for this derivation.
Thanks Inaki that makes total sense. I don't know where I got the idea that 1 private key => many public keys. I think because if you need to create a new address for every transaction, what links all your keypairs together to create one "wallet"? Why don't they say you need to create a new wallet for every transaction? – Ivan – 2016-11-25T12:39:39.577
1In the most basic form, a wallet is simply a collection of private keys that (eventually) contain unspent outputs for the corresponding addresses. When you want to send bitcoin, the Bitcoin wallet client then decides which of those unspent outputs to combine to send them. In fact you would only need to backup private keys for which there (still) are unspent outputs. But nowadays the private keys are typically derived using a deterministic method (BIP32) with a seed and since EVERY private key you use in your wallet is derived from it, you just need to backup the seed once and for all! – Bjarne – 2016-11-25T12:50:14.497