Call it what you want, it's still very wrong.
Not as rhetoric or metaphors.
Yeah, but if you understand it as rhetoric and a metaphor, you might be able to see where Ali was coming from, and that doesn't help when you're trying to craft a narrative of Ali as the bad guy.
When he spoke of being a slave for 400 years, the pugilist formerly known as Cassius Clay wasn't being literal or speaking of himself. He was speaking for the black man in general. Being an international sports sensation he was given a level of access to the media that someone who grew up black and poor in America was not usually afforded, so he used it to full advantage.
Faced with the truth that this country had indeed oppressed black men (and women, and children) for centuries, the racist (who insists he's no such thing) has to unpack the whole grim reality and attack it in parts in order to justify his disdain for Ali.
Was it exactly 400 years? No...therefore, to the racist, there was no oppression.
Did slavery last 400 years? Well if you count from the landing of the first colonists in the early 17th Century (ignoring the fact that slavery was a thing before then) and stop counting at the end of the Civil War, then it was only a mere 245 years, and that's not so bad, says the racist.
And the 100 years between the end of the Civil War and the passage of the Civil Rights Act? Why, that wasn't slavery at all! Says the racist, ignoring the century of institutional discrimination and Jim Crow.
That's how you get to the idea that Ali was a "black racist." Take his correct premise - that black people have been oppressed for centuries in this nation - systematically nit-pick it to the point where you can deny the obvious truth, and then take Ali's true statements out of context and claim he hates white people.