The specific reason that AMD cards are more efficient for Bitcoin mining is twofold and the answer to this question depends on both reasons:
- AMD GPUs tend to have lower-end ALUs but make up for this by packing more ALUs onto the card overall. For most hash algorithms this is preferable.
- AMD GPUs uses ALUs which can perform the 32-bit integer right rotate operation in a single instruction - Nvidia's ALUs take three processor cycles to perform the same operation. SHA256 uses this operation heavily, therefore Bitcoin mining uses this operation heavily.
Combining the two reasons above results in an overall 3x-5x faster generation of SHA256 checksums. However the two effects are not equal: Effect 1 leads to approximately 2x-3x Nvidia's performance all on its own, with Effect 2 accounting for about 1.7x performance advantage. Even if you were to choose a hashing algorithm that made less use of the BIT_ALIGN_INT instruction AMD cards would still be 2x-3x faster for hashing algorithms in general. This is because such algorithms are explicitly designed to be simple binary operations which AMD's simple ALUs can handle just as well as Nvidia's more complex ALUs.
In order to choose an algorithm which would perform better on Nvidia cards than AMD cards, you would not only have to eliminate scenarios such as #2 above where a specific instruction happens to be better implemented on one card than the other, you would also have to choose an algorithm with math complex enough that Nvidia's higher-end ALUs get the chance to individually shine. To my knowledge no such algorithm exists.
To back up my statements, here is some example data from oclHashcat's web site. The table below compares actual performance measurements from similarly priced AMD and Nvidia cards for the MD5, SHA1, MD4, NTLM and SHA256 algorithms, among others.

Keep in mind that in comparing PC1/2 and PC3/4 that the Nvidia cards actually have higher clock speeds in both cases and still perform worse. In no case did Nvidia's performance approach the performance of a similarly priced AMD card, and to my knowledge no currently implemented algorithm performs sufficiently complex mathematics to make use of the more advanced ALUs on Nvidia's cards.
In short, Nvidia cards are like a small room with several fairly intelligent people, each capable of complex mathematics while AMD's cards are like a sweatshop. Hash algorithms are the computing equivalent of mass-producing name-brand tennis shoes and so the AMD "sweatshop" card will always outperform.
5The bitcoin protocol should not be changed to make NVIDIA happy. – eldentyrell – 2011-10-16T19:25:34.347
1Agreed, I was just wondering if an alternative currency could be designed to use NVIDIAs. – ThePiachu – 2011-10-16T19:59:35.620