The benefit of hardware deterministic wallets is that your keys are permanently kept offline and secret, as long as you don't leak your seed words. To spend bitcoin, the transaction signing is performed on the device, keeping your keys safe.

If you encrypt the keys yourself (I'm assuming you're talking about bitcoin private keys, not an HD wallet seed), it depends how you store them. I can see three risks:

1. If you're planning on storing them online and encrypt them all with the same passphrase, then you're reducing the security of your keys to the single passphrase. If you leak your passphrase, anybody with access to your data can decrypt and use your keys. If you don't store them online (including on a system with access to the internet) however, this is less of a risk.
2. Signing transactions. In order to sign transactions, you're going to have to decrypt your key and run it through some software. If you do this online, it's very risky. If you do it offline, you're still at risk of keyloggers and other malicious software that could be present on your machine. This also applies to when you generate your keys in the first instance.
3. Loss. If you lose your encrypted data or your passphrase, your bitcoin is lost. With an HD wallet, if you lose the wallet you can recover with the seed words. If you lose the seed words you can still use the wallet to transfer to an alternative address.

If you do decide to use the passphrase method, note that there is already a standard for protecting a private key with a passphrase (example implementation - check the paper wallet section), so it's not necessary to encrypt it separately yourself.

Hardware wallets are preferred to hot wallets and cold storage, as they cannot connect to the Internet, and do not require key input into a computer in order to sign transactions. The second point is important, but also partially wrong. The hardware wallet itself is a computer with very specific set tasks, mainly, key pair/seed generation and transaction signing. The keys are never supposed to leave the device and never were outside of the device, to begin with. You cannot input keys manually into a hardware wallet. They have their own random number generator hardware, which feeds into the CSPRNG, then the key/seed generator. This means, incidentally, that you do not need a "hardware wallet" to have a hardware wallet. An air-gapped computer put inside a dark room, with all the source code verified by hand, in a Faraday cage, and a whole lot of other security measures is also a hardware wallet. It's just much simpler to have a dedicated device engineered specifically for that purpose.

Online computers run the risk of the plaintext private key (or even the encrypted private key) being salvaged by malware. Once outside of a sandboxed and enclaved environment, always outside a sandboxed and enclaved environment.

The only attacks on hardware wallets require direct interfacing, proximity, or some zero-day that utilizes an infected user's computer to send commands to the wallet (e.g. load a custom firmware that makes key extraction possible). Direct interfacing requires that the hardware wallet is stolen, but will not necessarily output any keys/seeds, since ECDSA is resistant to chosen ciphertext attacks for the most part. This would also require a zero-day to directly extract the keys. The proximity exploit would require access to the environment surrounding the key in order to influence the random generation, making the result more predictable and easier to attack (even this is a stretch, as they should have RF shielding). Given how limited these attacks are, unless a hardware wallet contains fortune, it's generally not worth an attacker's time to attack a hardware wallet from far away. It's much simpler for them to attack exchange wallets or hot wallets of normal users.

Note that none of these methods are secure against brute force. There is no way to stop brute force, it's just extremely inefficient. However, it has worked before.. Brute force aside, an exchange wallet is the least secure, a hot wallet is still not secure, a hardware wallet is secure, and a cold storage wallet is impossible to attack electronically altogether.