You're basically correct. The rate at which block reward and thus inflation (or lack there of) is determined is by block height alone. That is, starting from a maximum of 50 new bitcoins generated per block at height 1, and halving that amount every time 210000 blocks have been generated, the maximum amount of new bitcoins that are allowed to be generated by a generation transaction in a block eventually becomes less than one satoshi, which is the smallest unit of amount of in a coin, at which point generation of new bitcoins stop.

On the other hand, the difficulty, or technically the "Target" that a block header hash must be below (it's just a big number) is only determined by the rate at which blocks were generated during the previous 2016 blocks re-target period, starting from the very first and highest target which is also the lowest difficulty (difficulty 1) onward. This means that difficulty (or target) can change and does go up and down as hashrate enters and leaves the network.