Moving from subtitles to native content—is it too early?

I’m a high-intermediate learner trying to ditch English subtitles on Latvian shows, but I find the rapid, connected speech almost impossible to parse. Is there a better approach to listening than just 'listening a lot'? I want to move toward full immersion, but I feel like I'm hitting a wall where I understand the words but not the flow.