It's easy to say that as a white person. Not so easy when you're a non-white person and have to deal day in day out institutional racism that is still in effect.
The saying "Check your privilege" exists for a reason.
Ah, yes, family history. My mother was called "That Jewish Girl" in a small town where there was exactly one family with Jewish heritage. I, myself, have had one case of racism, myself -- I disclosed that my grandfather was born Jewish, but he married a Baptist and converted. Immediately he left my porch (we had been drinking beer and watching our 5-year old girls play) and refused to allow his child to play with mine ever again.
You're right, though. One incident was quite hurtful. But I don't wear my heritage on my face. For those who do, oh my. No need to mention "black" heritage, or in a prior era Italian or Irish heritage, when your features tell the tale.
We are endowed by genetics with the ability to categorize by group. To recognize close relatives as close relatives and to favor them -- kin selection. Quite automatically we group into us and them. It is natural to identify with kin. Black brothers and sisters. White brothers and sisters. Daughters of the American Revolution.
It is as unnatural as a tenor singing soprano to fight this tendency. It can be done but it is fighting biology. We are a tribal species. Do unto others within the tribe as you would be done by. Distrust outsiders. This is biology.
The world will be a better place when tribalism is limited to warriors engaged in non-fatal combat: sports.
Can't we all just get along?