Struggling to remember noun classes—is there a pattern or just brute force?

I'm a grammar-focused learner, and the Wolof noun class system is absolutely destroying my confidence in sentence construction. I understand the concept of vowel harmony in some contexts, but trying to memorize which prefix attaches to every object is slowing down my speech to a crawl. Does anyone have a mental framework for how these classes group together, or is it purely rote memorization?