Most likely it is a small design error, but there is big argument about it at bitcointalk.org:What would you change about the Bitcoin protocol?.

Modern computers almost always use little-endian internally, so this choice improves speed. If Bitcoin used network byte order, then order conversions would be necessary for every message sent or received.

Hashes are defined by the standards as being big-endian, and crypto libraries deal with them in that form, so hashes are transmitted in big-endian. Bitcoin displays hashes in little-endian because Bitcoin sometimes considers hashes to be little-endian integers instead of strings.

Bitcoin transaction and block hashes are considered little-endian when treating them as integers. On the network they are just 32-byte sequences in the order they are generated by the hash function.

In both transaction and block hashes bytes are reversed when displayed by Bitcoin software and websites in hexadecimal form. This makes blocks show up in big-endian form, which is the way we humans write numbers down.

I would suppose that the idea of reversing bytes comes from the fact that block hashes in Bitcoin are treated as numbers for the purposes of difficulty calculations, meaning the hash must be below certain threshold in order to pass. It is natural for a human to expect such a number to be displayed on screen with leading zeroes. However this is not a very solid reason, since block hashes could have been interpreted as big-endian when converting to integer, while paying a tiny performance penalty.

As for transaction hashes, I have no good explanation as to why they are also reversed. Perhaps it is just for consistency.

Modern CPUs also typically have byte-swapping load and store instructions, or at least a byte swap register instruction, so that effective programmers (and optimizing compilers) may eliminate, or nearly eliminate, the overhead of endianness compensation.

This "small design error" may be a simple case of "premature optimization".