Here's a short explanation that might help you understand it. It's grossly oversimplified but just designed to help you get the general idea.

Imagine a world in which multiplication and addition were irreversible. That is, division and subtraction do not exist. It is not too difficult mathematically to create a world where division and subtraction are trillions of times more difficult than multiplication and addition, so for practical purposes, division and subtraction are impossible.

Now, let's say we all agreed on a very large number. Call it G. It's some big number we all agree on.

Now, you can pick a random number and store this someplace very safe. Call it your "private key", S (for secret). Now, multiply it by G. Call the result your "public key", P. You can tell anyone your public key. Since they can't divide it by G, your private key will stay private.

Now, say you want to "sign" something, O. What you do is you take that something, add your secret key to it, and publish the resulting number. So you publish: O+S, that's your signature

First, nobody can tell your private key from that number. They'd have to subtract, and that's not practical. And nobody else can produce that number, they'd have to know your private key to know what to add to O.

So all we need is for someone to be able to prove that's the "right" number and we're there -- and they can do that. Why? Because G * (O+S) = G * O + G * S.

They know what you claim is O+S ... the signature. G*S is your public key. And they know O. So with just some addition and multiplication, they can compute both G * (O+S) and G*O + G*S. So they can check if the number you published really is O+S if those two are equal, thus that the person who computed it knew your private key, and thus it functions as a signature.

The challenge, of course, is to create systems in which multiplication and addition are possible but subtraction and division are not. And there the math gets complicated. Bitcoin uses a system where division requires discrete logarithms and a change to the algorithm so that it doesn't matter if addition is reversible.

Wikipedia is a wonderful source for this. First, note that there is no encryption in the Bitcoin protocol - there are techniques from the more general field of cryptography, such as digital signatures and hashing.

Digital signatures are an example of public-key cryptography, which can be used not only for digital signatures but for a type of encryption which is different from traditional symmetric encryption.

Specifically, Bitcoin uses Elliptic Curve DSA for digital signatures and SHA-2 for hashing - here is a nice review of how they are used.

You might also want to read up on RSA which is a very well-known public-key cryptosystem, though not used in Bitcoin.

Small clarification - encryption is generally a topic about obscuring data so it can't be read. If you are interested in the algorithms of signing things, you are not looking for encryption, but you are looking for signature algorithms.

Signature algorithms are generally a topic about mathematically showing that only you could've sent the data, while not necessary obscuring it.

For the signature algorithms used in Bitcoin:

1. Google for ECDSA.
2. Wikipedia entry for ECDSA.
3. Bitcoin Wiki entry for ECDSA.

Generally, look into topic of Digital Signature Algorithms in general, and Elliptic Curve Digital Signature Algorithm in particular.

This topic is a field of cryptography, and is not Bitcoin specific. If you have more questions regarding it, you should check out Crypto SE.