That's the idea behind the concept of infinite regress. Time has no beginning. It extends into the past for an infinitely long period of time.
But as I've said this idea makes no sense. If infinite time must pass before yesterday occurs then yesterday will not occur.
Calculus describes very logically how rates of changes work in a continuum, and we know rates of changes occur in reality. So if we assume that we are in a continuum, then we can let calculus take the wheel.
By assuming a continuum, there is an infinite number of moments that pass in an hour, second, day, etc.
Since that we may actually be in this continuum, one cannot say for sure if our days are infinitesimal to infinity many days in a finite subinterval of a larger interval.
Your logic must at least assume that there is no space-time continuum.