Ok, so let's take your simple example and add a few details to it. We have 100 people in society, 3 of them are slave owners, 18 are slaves. One slave owner owns 12 slaves, another owns 5, the last one owns 1 - closer to the real distribution it was back then.
The vast majority of contribution of the pie of the 18 goes to those 3, and very slightly to the other 79 non slave(r) population through lower prices of whatever the slaves make.
Now, the question is whether society as a whole benefits from the slave situation like you claim from the standpoint of the non-slaves and their decedents.
Economics 101 says that labor and capital are substitutes - the cheaper the labor, the less investment in capital - if I have a dollar to spend and I need to increase production, I can spend it by buying more labor or I can spend it by investing in capital. If one dollar buys me more labor (slavery) - then I allocate more dollars to it than capital investment. As a result, capital investment in the country is less. The other 79 therefore become slightly less productive (and earn lower wages) as a result of the diminished capital investment.
Furthermore, what about an alternate situation where there were no slaves, where those individuals are free to obtain an education and develop more skills and work freely for themselves or anyone else? Think of the contributions they could make that would benefit society as a whole. Surely some of those now free slaves would contribute good ideas and or discover something (whether it be new business processes or academic research) that makes the economy and society just a little bit better. In other words, they would contribute to economic growth much more so than they did while slaves. The pie is smaller since they are slaves.
Now let's fast forward to the situation when slavery is outlawed. Our economy has a smaller GDP than it would otherwise have for the reasons above, and the 79 non-slaveholders and their families are all worse off because of it - less tax revenue to support government services, slightly lower wages, and lower capital stock. No question that the 3 slave holding families benefited, but are you really going to argue that their benefit outweighs the loss to the 79 non-slaveholders (to say nothing of the damage to the slaves themselves)?