Technically no, you only need to know your UTXOs so you can make a new transaction and sign that yourself. But if you want to use a Qt GUI or similar, then yes, it will attempt to download the entire blockchain again I would imagine.