Do I really need to master Zhuyin if I only care about speaking?

I’m a traveler planning to live in Kaohsiung for a few months, and I want to focus purely on listening comprehension and speaking. Some locals tell me I don't need to learn Zhuyin because characters are enough, but I'm worried I'll miss out on the nuances of Taiwanese Mandarin pronunciation if I use Pinyin as a crutch. Has anyone had success getting to a conversational level in Taiwan while skipping the script, or is it a major mistake?