As far as productivity and wealth to the particular nation (in our case the US), it really does not matter whether the workers are replaced. As long as the factory stays in the same country the productivity and (wealth if you will) stays in that country.
But here is that thing that is key. Once that wealth stays in that country, there is a good chance that it will create other support jobs. There is no chance of that at all if the factory moves to China.
Your math is wrong. You are discounting the possibility of the workers who lose their job getting replacement jobs, even if those jobs are lower paying.
Let's say a US company produces 1,000 tons of steel that sells for $10,000,000
It pays wages of $3 million to produce the steel.
It decides to relocate the factory to China. Now it pays wages of $500,000. The price of steel drops to $9,000,000 as more factories relocate to China and the competition and lower costs leads to lower prices. In other words, the wealth of steel consumers goes up $1,000,000 (actually a little bit more, as new buyers will enter the market due to the lower steel price, meaning steel production and consumption will increase a little bit and there is new wealth via consumer surplus created from that, but we'll ignore that for this example).
The company keeps US jobs with wages of $500,000 to handle import/export and other international supply chain issues.
So the company now has $1,000,000 additional profit with the move ($2,000,000 in labor savings minus $1,000,000 less revenue).
So, on the plus side, we have $2,000,000 increase in wealth.
However, you'll point to the $2,500,000 lost wages in the US and say that means the move actually decreased US wealth by $500,000. But that would only be true if none of those who lost their job are able to find a new job.
We know that is not the case. What happens is that these workers are often able to find other jobs (especially if they are willing to move to a different city), jobs that do pay a bit less, yes, but jobs that produce wealth none-the-less.
Let's say that 10% of those workers can't find a replacement job or find a way to retire early. The other 90% do find another job. Let's say that the new jobs pay 25% less wages/benefits on average.
Now, we have those workers earning wages of $1,687,500 at the new jobs.
Now when we add up the totals, we have wage decline of only $812,500 ($2,500,000 - $1,687,500). Add the $1,000,000 in company profits and $1,000,000 in consumer savings, total wealth increases by $1,187,500.
On top of that, the money paid by US customers for the Chinese steel tends to make its way back to the US. This is because when the steel is imported from China, US dollars are sold on the international market to buy Chinese currency to pay for the steel (or, in the case of a multinational company, to pay for the expenses of their Chinese factory to produce the steel). When those dollars are sold, who do you think buys them? Someone looking to spend that money in the US (after maybe going through a currency dealer, the dealer then wants to find a buyer for those dollars). The buyer is looking to either buy US products/services to import into their own country, or they are looking to buy US investments (stock, real estate, bonds, company expansion for their own US branch of an international company, etc.). Thus, you can't discount the effects of a significant portion of the money coming back to the US either and the effect that has on our export industries or our capital stock.
Finally, the 25% wage decline for the workers who find a new job is a short-run effect. In the long term, those workers will retire and the new, younger generation will enter into areas of the economy, at a young age, that is more thriving and has the potential for higher wages over time. These jobs that are getting lost to China are stagnant areas of the job market. Putting things like tariffs or other restrictions on them is trying to preserve the jobs of yesterday rather than preparing/adapting to the jobs of tomorrow.