I may be able to explain that. At least, I can explain a similar disconnect that I've experienced.
All the little "mistakes" you obsess over while you're speaking? The 5-second pauses to collect your thoughts, the occasional "um" or "uh", a bit of restless hand wringing or fidgeting?
The audience doesn't care, or even notice 99% of the time. And by the time you start talking again, they've already forgotten.