No matter which way you look at it, the issue ends in an absurdity. If the past is infinite, then it seems hard to fathom how it would have ever become the present. Yet, if the past is bounded, we have to claim there was a first event, itself uncaused, that set everything in motion.
Both objections can be mitigated somewhat by treating time as a dimension just like any of the spatial dimensions. The universe would thus be a large object with x, y, and z dimensions that we can visualize roughly as a 3-D structure, but also with a t dimension representing time. At every point on the t axis (which we are unable to visualize) is a slice of the 3-D universe at a particular moment. One of those slices corresponds to what we call the present while the others correspond to either the past or the future. But those labels are a consequence of our subjective perception of the flow of time; in reality, all points on the t axis are equally prominent. Just as there is no center of the universe in spatial terms, there is no privileged coordinate in temporal terms.
Having reduced time to a dimension like length or width, the question of an infinite past becomes no different than the question of infinite space. If someone suggested the universe was infinitely wide, we wouldn't say "that's impossible, because then someone would have to traverse an infinite distance to get here." This is because we do not experience space as constantly advancing as we experience time. So, I think an infinite past is not refuted by the objection that an infinite number of moments would have needed to happen before now, as if now would somehow "never arrive" in that case. But if time is just another dimension, there is no privileged "now" that all past events are striving towards--it's just points on a number line.
This doesn't even get into the possibility of time curving back on itself, as has been suggested for space in a number of different manifolds. So, I think the issue is simply not something we can solve by logic alone, since the universe at its extremes may be illogical.