There's a bigger attack surface for someone that wants to doxx you. They only need to link a single transaction to your real life person to know all the transactions you've participated in. A single payment to newegg, for instance, will identify you across all transactions.
Very few transactions are completely anonymous. But yes, if they're all completely anonymous, then you won't be hurting your privacy. However, that's much more the exception than the rule and you're probably leaking much more information than you think you are.