Moving from subtitles to native content: The 'pretérito perfeito' wall

I’ve reached a plateau where I can understand most B1-level Portuguese content with subtitles, but as soon as I turn them off, I lose the thread completely. I think my brain is still too focused on translating the irregular past tense verbs instead of just absorbing the flow of the language. How did you guys manage to move past this plateau and stop relying on visual aids when listening to Brazilian podcasts?