I've never been worried. It's actually quite a bit of work to risk a rape accusation, while it's quite easy to avoid the problem, but still get plenty of sex. Having sex with an uncooperative woman is difficult. It's not so easy with an unconscious one. If a man prefers either, that's a fetish, and will eventually lead to a lot of trouble.
My standard advice to young men who feel they aren't getting the kind of affection they think they merit, has always been, "Clean shirt, clean fingernails, smell good, and smile." That will get you in the door, and maybe into bed. Leave anything off the list and you don't have shot.
If one wants to avoid being slapped with an erroneous rape charge, it's really simple. Only have sex with women you know and trust. If you don't know any women, or don't trust any, your way may be best.