How do you handle the -ka/-krap particles in casual conversation?

I'm a heritage learner and I grew up hearing my parents use polite particles, but when I try to use them with my cousins back in Thailand, I feel like I sound way too stiff or overly formal. I want to sound more natural and less like a textbook, but I don't know when it's okay to drop them entirely without sounding rude. Is there a rule of thumb for when to switch to casual speech versus keeping the formal particles?