Why is the instrumental case suffix so inconsistent with feminine nouns?

I’m hitting a wall with the instrumental case in Slovak. I keep trying to memorize the endings, but every time I think I’ve got it, I accidentally use the accusative form instead. Does anyone have a mnemonic or a specific way to visualize the shift from 'a' to 'ou' without constantly stopping mid-sentence to calculate the gender?