How do I fix my 'virile vs. non-virile' gendered plural mistakes while speaking?

I’m a heritage speaker trying to connect with my grandparents in Poland, but I constantly default to using the wrong gendered plural endings for groups of people. It’s embarrassing when I mix up 'oni' vs 'one' when talking about my cousins. Is there a trick to getting these gendered plurals right on the fly, or should I just focus on memorizing the patterns through more input? I am asking specifically about learning Polish, not a generic study routine.