An exoplanet that has an atmosphere of 78% Nitrogen and 20% oxygen could easily be uninhabitable by humans if the remaining 2% were to include a certain amount of chlorine, for example. Or of carbon monoxide. Or of a fairly large range of organic poisons, that might hypothetically be deployed by the local biology as a competitive edge over evolutionary rivals.
The terrestrial biochemistry to both produce and tolerate oxygen was a major boost for the species that first evolved that pair of abilities, and oxygen production now dominates the plant kingdom and is common in such widespread life forms as phytoplankton; while oxygen tolerance is almost completely universal amongst terrestrial organisms.
An exoplanet on which Cl2 production and tolerance were similarly widespread, would be a very unpleasant place for humans, and for their crops and livestock.
Are any of those stable, though? Most of the nasty stuff is rather reactive, would it stay in the atmosphere if something widespread wasn't continually producing it? Even oxygen can only exist in the atmosphere due to production, it is eliminated by reaction but slowly enough that the vast quantities pumped out by photosynthesis can maintain the supply of oxygen.
The non-reactive toxic stuff isn't exactly prone to sticking around in the atmosphere.
You answered your own question.
Terrestrial life has evolved to produce a pretty vicious poison - oxygen - and to tolerate it. As a consequence, the Earth has a very unusual atmosphere, which contains almost 20% of a molecule that is only present because of terrestrial life.
An exoplanet without life would not be expected to have an oxygen rich atmosphere; Conversely, an exoplanet with alien life could easily have an atmosphere that is rich, not only in oxygen, but also in other unstable toxic chemicals, such as (for example) chlorine.
Such an atmosphere would be inimical to terrestrial organisms, despite its oxygen content.
There's no good reason to expect any exoplanet to have an atmosphere that humans can breathe. Our atmosphere is basically a part of the global extended phenotype; It's a reflection of the biosphere that maintains it in its current composition - and it has changed repeatedly over time as various biological factors have come and gone.
Right now, humans are pushing up the carbon dioxide level. What effect that will have on our species survival remains to be seen. But there's no doubt that a hypothetical time traveller could go to Earth's past and find the air unbreathable, and no reason to expect that the same won't be true at some (hopefully distant) future date. And if the very atmosphere shaped by Earth-like organisms is only occasionally able to support H. Sapiens, the idea of an exoplanetary atmosphere being able to do so, at any random time in its lifecycle, is absurd.
To travel for decades through the instantly lethal void of space would be bad enough, but even worse for any proposed colony is to get to your new outpost, only to find that you're
still confined to (and utterly dependent upon the ongoing integrity of) your spacecraft or cumbersome suits, because there is no breathable air.
And you need to enclose your crops and supply them with earth air, and earth soil, because the local soil bacteria would rapidly poison the air if you used it.
Sterilising megatonnes of topsoil and then seeding it with terrestrial microbes and other organisms sounds like a lot of hard work. Probably better to just stick to trying to colonise Antarctica or Siberia.
