First of all, many trafficking victims are employed in industries other than sex industry. Should we outlaw those too?
Second, anti-trafficking groups tend to conflate all sex work with trafficking. Thus their figures are unreliable.
Third, rape requires intent. Let's say an Icelandic woman's kid was kidnapped and the kidnapper ordered her to seduce you and sleep with you or her child will be killed. Did you just rape her?
You have no way to ascertain if the prostitute that you rent is a slave, child slave or just loves to fuck men that are unable to get a woman without payment.
Neither do you have a way to ascertain for sure whether the woman you sleep with and are not paying is acting under some duress like in the scenario above. If we are going to go by hypotheticals, there are many of those.
However, there are ways to reduce the chances greatly. I prefer independent contractors and also have a couple of regulars whom I have gotten to know relatively well.
It is like with stolen goods, you may say that you did not know that the camera you bought from a guy on a street corner was stolen but the court will say that you should have suspected that it was.
Except that I am allowed to go to a store and buy a camera.