BTW, not all means of time travel have this problem.
Robert L. Forward,
Timemaster has time travel via wormhole. The wormholes are physical objects in space, moving them around does not change the linkage between them. (In fact, when created they are not time travel devices at all--the time travel aspect comes from hauling one end around at relativistic velocity.)
While there is a certain amount of handwaving to create the basic technology he was a hard-science author and apart from the negative matter & wormholes I can't find any scientific flaws. (And, yes, that includes the stardrive he uses to haul them around.)