Just a quick mention..... Probably not the issue, but when I swapped a 350 into an '80 el Camino that originally had a 305, I had what sounds like a similar problem.
Seems I forgot to switch the flexplate, (I was young once... and dumb..) and that was what was causing the binding as I reused the 305 starter.
Two broken starters later, I pulled the transmission and put in the right flexplate.
But, like I said, if the drivetrain has never been apart, it's probably not the issue.