The higher difficulty means you will be reporting results less frequently to the pool. This reduces network load on both your system and the pool. It also reduces the restart delay for your mining hardware as it prepares for the next work unit. Most pools base the rewards on the number of difficulty 2 shares accepted. So they increase the reward based on the mining difficulty. If you had four shares accepted at difficulty 4, you would be credited with the reward for 8 shares. Similarly, four shares at difficulty 8 would yield a reward for 16 shares.

So it is to your benefit to set the difficulty high enough to reduce the network and mining delays but not so high that you get a lot of stale shares. Some pools (such as BTC Guild) will automatically adjust the difficulty based on your hash rate.