The number of accusers is a bit of a red herring; what matters is the strength of the evidence. Is there corroboration? Certainly there have been instances where there is no accuser - the victim was murdered, but there is physical evidence of sexual assault. As is oft stated, claims made without evidence can be dismissed without evidence. To leave it just to the accusation, without more, is to return to witch trials.
Except we have evidence, just not direct evidence. Ford made a bunch of statements about the situation which were able to be checked. While they don't prove an assault they prove her story is not simply made up. For her to have remembered those details makes it pretty certain something very memorable happened at that party.
Ordinary parties rarely include highly memorable things outside the sexual realm--assault and losing one's virginity are the two biggies. If it were the latter her partner would have no reason to deny it.