The ledger has to be public in order for Bitcoin to be decentralized. The designers of Bitcoin didn't want to need anyone to play the role of a "bank" who keeps all the records, and whom everyone else has to trust to do that properly.
For instance, when you keep a bank account, you are trusting your bank to keep your money and give it back to you on demand. There isn't anything that actually prevents them from just taking your money and faking their account records to make it look like you never deposited it. You could fight them in court to get it back, but they still have your money in the meantime.
But there still have to be records, and someone has to be able to see them to verify that they make sense. Bitcoin's solution is to make the records public so that anyone can verify them.
You're right that this comes at a cost in privacy. (Though from a certain point of view, in the current banking system, your bank has all your information; you're trusting them to keep it private. If they published all your bank statements in the newspaper, you could probably sue them, but the information would still be out there.)
But your "coffee shop" example isn't a very good one. Of course, if they operate with a single address posted on the wall, then others will be able to see their incoming transactions. If that's a concern, then they shouldn't use that approach; they should use a new address per transaction, as you suggest. That shouldn't be very difficult. The coffee shop is going to have to have some sort of Internet-connected device (computer, tablet, etc), in order to see whether customers have actually made their payments and can be given their coffee. If so, then that device can easily generate a new address for each customer, perhaps displaying it as a QR code on its screen.
2"which would want only a single address for people to send money to."
No, they wouldn't want to do that. They'd want to give a new address out in each payment request. Which kind of already solves your question. – Murch – 2016-06-17T08:10:41.903
It does complicate things a lot for a small shop though. They'd have to generate a new address per transaction, and display it to the customer. The proposed solution of a QR code on the wall is a lot simpler to adopt. Tradeoffs... – user36303 – 2016-06-17T08:14:41.040
Having customrs scan a QR code on a screen in front of the counter is just as easy. Compare that use case to using credit/debit cards, where cards must be swiped, chips read, PINs entered, and receipts signed. A screen showing a new QR code for each customer is way better. – Jestin – 2016-06-17T13:20:18.637
@Jetsin the problem is reliance on everyday technology, which can fail. In a credit card, if the scanner fails, you can just read the 15-digit decimal number off the strip and enter it manually. This is much more annoying to do with a 64-digit hex number (try it yourself). Also someone - say my aunt - who is unfamiliar with technology might find it very unintuitive to scan a QR code to pay for coffee. For the Bitcoin process to be widely adopted, it must become stupid to do wrong. – Abhishek Divekar – 2016-06-18T05:11:57.420