Why do I keep forgetting my classifiers when speaking?

I can read Vietnamese text pretty well, but the moment I try to speak, my brain freezes when I have to choose between 'cái', 'con', 'chiếc', or 'bộ'. Is there a mnemonic or a logical rule to stop stalling every time I need to point at an object, or is this just something that only brute-force memorization can fix?