A human driver processes inputs (sight, sound, touch, smell, and taste); processes those inputs, to determine what needs to be done; and then makes the appropriate changes to the controls (steering, brakes, throttle etc.).
A software agent doing the same task has more information, and obtains that information faster than a human driver - it can have visible input from cameras mounted so as to eliminate blind spots, plus it might also have radar, infrared, information from other vehicles nearby, information about the precise torque at each wheel, etc. etc.; It can process that information more rapidly than a human, and without boredom, distraction, or degradation of performance due to drugs, alcohol, or fatigue; and it can exercise more precise control than a human, potentially directly adjusting any aspect of the vehicle and it's engine/motor thousands of times per second to achieve the optimum results.
In principle, a self driving car should be better able to handle poorly maintained roads than a human driver. The problems with current self driving cars is that they are not programmed to be sufficiently gung-ho and stupid to do the things that humans regularly do, that are mostly not fatal by good luck, rather than good judgement.
A human driving on a poorly lit road at night in a downpour will guess that the road he can't see carries on straight (or follows the contours of the land around a curve he also can't see), and might well get it right 99.9 times out of a hundred. A self driving car will calculate the odds of a crash at one in a thousand, declare that an unacceptable risk, and do what a human driver SHOULD have done, which is to not proceed until it can be sure of where the road is. Humans don't like that. They prefer to get there, even if the cost is that once every couple of decades they end up crashing off the road into a ditch.
Google's cars are bad at handling four way stops, not because they don't have the appropriate rules for such junctions, but because nobody else on the road obeys the rules. If four Google cars arrive at such a junction, they all stop, then all move in turn, based on the road rules. But if a human driver is present, he edges forward, trying to bully the other motorists into giving him a right of way he technically doesn't have; the Google car is 100% always intimidated by this (because it is programmed to always give way to any moving vehicle in the junction), so it never gets through until all the human drivers are gone.
Human drivers are arseholes, and try to bully each other. The thing that stops most of them from doing this constantly is that the behaviour is usually ineffective - the driver they are trying to bully often refuses to yield, and it becomes a game of 'chicken'. With the bully in the position where he stands to get a dent AND a ticket, while the guy with right of way only stands to get a dent, the bully is best served by backing down. But when the car is programmed to avoid collisions at all costs, the bully knows that neither dent nor ticket are possible - the software agent will always cave rather than take the slightest risk.
What self driving cars need, if they are to share the road with humans, and not get intimidated, is to grow be programmed with a pair.
The most important optional accessory for a future self driving car will be a robot arm that can, when circumstances dictate, extend from the driver's side window, and give the finger to other motorists.
Self driving cars will be very effective, once the software is able to make the very difficult distinction between being a defensive driver, and being a total pussy.