The signatures in Bitcoin (and Bitcoin Cash, Ethereum, and many other cryptocurrencies) are generated using elliptic curve curve cryptography. I found these videos very helpful in that regard: https://www.youtube.com/watch?v=jQLmaBjYrk4

Other than that, you probably want to look into hash functions and asymmetric cryptography in general.

As a general introduction to cryptography, I took the cryptography course on Coursera which I can really recommend. This goes into the role of the adversary and what he can achieve. The teacher of this course, Dan Boneh, is, I believe, among the ten most renowned and best cryptographers in the world. On top of that, he is a very good teacher. The course does go into both symmetric and asymmetric cryptography but in the asymmetric field, it only teaches you RSA and Diffie-Hellman, not elliptic curves. Use the above videos for that.