The entire US economy? That's a bold statement. Do you have evidence for this?
...
In those days many of the rich were those who dealt in cotton and slavery
This was a pre-oil, pre-Industrial Age society.
Innovation was fueled by the wealth created in the cotton or slave trade.
All in a society benefit from the innovation made within the society.
This claim intrigued me so I did a bit of research. Results aren't conclusive, but still might be interesting.
Starting with an established list of the wealthiest Americans (relative to the U.S. GDP of their time), there are 12 names who died before 1860. (The top three on this list were much wealthier than the other nine.) All but three of the men were largely self-made.
John Jacob Astor 1763-1848 :
. . . . New York, fur and opium trading, real estate
Stephen Girard 1750-1831 :
. . . . Penns., shipping, banking
Stephen Van Rensselaer 1764-1839 :
. . . . New York, landowner -- inherited
Elias Hasket Derby 1739-1799 :
. . . . Mass., shipping, privateering
Israel Thorndike 1755-1832 :
. . . . Mass., shipping, privateering, industrialist
John Hancock 1736-1793 :
. . . . Mass., merchant -- part. inherited
George Washington 1732-1799 :
. . . . Virg., real estate, planting (tobacco) -- inherited
Thomas H. Perkins 1764-1854 :
. . . . Mass., shipping (slaves, furs, opium)
John McDonogh 1779-1850 :
. . . . Maryland & New Orleans, shipping, real estate
Samuel Slater 1768-1835 :
. . . . Rhode Island, textile production (cotton)
Benjamin Franklin 1706-1790 :
. . . . Penns., publishing
Peter Chardon Brooks 1767-1849 :
. . . . Mass., insurance of ships (mostly slave-traders' ships)
The only planter was Washington, and he planted tobacco. Only one of the names is associated with cotton, and he operated textile mills in New England. (There are few slave-owners on the list; those few freed at least some of their slaves.)