I was always under the impression that the Tower of Babel, when completed, was a structure so massive and so grand, it defied human comprehension to the point people went mad and began speaking in strange tongues.

Essentially, a man made eldritch abomination.

I don't think God was actually afraid that people would be able to reach him as much as it was that he just took offense to it. There are a few ways that the passage can be interpreted, but knowing that God was, well… God, I don't think he was legitimately afraid of any consequences. He just needed to show them their place.

Then again Jacob did wrestle with an angel until dawn so idk