1
My understanding of "hardfork" was that it referred to anything that requires re-building the blockchain from the genesis block. But now, I understand that anything that requires a Bitcoin-Qt client upgrade is a hardfork, too. Why is a hardfork considered a problem in the Bitcoin community?
User wallets and stored keys are more important, so all upgrades must ensure they wouldn't break these, but does it matter at all that it's required to upgrade old clients to use new features?
2
All the nodes in the Bitcoin Network have agreed on a certain set of rules. These rules are coded into the client and everything that changes the state of the ledger is checked against those rules.
Hardforks happen when a change in those rules (for example the reward halving rule or the difficulty adjustment rule) causes the new clients to make different decisions about the validity of transactions and blocks. This basically means that over time the state of the ledger on older clients would diverge from the state at new clients, creating two incompatible networks, as transactions would be valid in one but not in the other.
As transactions and blocks are chained together, even a single disagreement about the validity of a transaction or block can cause large parts of the state to become inconsistent over time, it is therefore crucial that the network as a whole agrees on the rules.
The developers keep a wishlist of changes for hardforks, because hardforks should be a rare event, breaking changes be bundled into on hardfork. The changes decided in hardforks have a trigger height after which they are applied, which is usually several months in the future in order to let developers of alternative clients and the users update their nodes before the switch, minimizing the number of incompatible nodes left behind.
1
A hardfork is not when old clients must upgrade to use new features. A hardfork is when old clients must upgrade to use the system at all. For example, if a new type of transaction was added, people who didn't upgrade wouldn't be able to understand the new transactions and wouldn't be able to spend or receive any Bitcoin that had ever been transferred by a new transaction anywhere in its history.
1This does make it sound like only a disagreement about the effect on the ledger is a problem. Even something as simple as changing the encoding of digital signatures could however cause a hard fork, and it's not just "large part of the state" that become inconsistent: it is immediately all of the state that starts diverging, as the two sides do not accept eachother's coinbases and transactions that depend on it. – Pieter Wuille – 2014-02-09T14:35:46.320