Americans living in other countries would be royally screwed if it didn't work this way.
America is, I believe, unique in that she taxes the income of non-resident citizens. Literally no other OECD country does that.
Not only that, she also taxes non-resident non-citizens on income "earned in the US" - I have never lived in, worked in, nor been a citizen of the USA, but because a previous employer gave me some shares as a bonus, and these shares pay dividends via the NASDAQ, through a broker in New York, I pay US taxes (which I can offset against my Australian income tax) on those dividends.
Meanwhile I retain UK citizenship, and pay absolutely nothing in UK tax (other than VAT on any purchases made when I visit family over there - and for any such purchases that I subsequently export to Australia, I can claim that VAT back at the port of departure). The sum total cost to me of my ongoing UK citizenship is the fee for renewng my passport every decade, and that is a voluntary expense - if I don't plan to use the passport for a while, I can let it lapse and renew it at some future date of my choice, or not at all.
Americans living in foreign countries shouldn't have to deal with the IRS at all. I certainly don't have any dealings with the UK tax authority - I am not even sure what it is currently called, as I believe they changed the name more than once since I emigrated.