Sample size isn't magic and does nothing to fix the flaws in method. It only overcomes problems of avoiding patterns that emerge by random chance due to sampling error. And it is a basic fact of probability that an increase in sample size has exponentially diminishing benefits once you go beyond a couple hundred observations. The one study I referenced sent out a total of 1374 resumes, which given the absolute massive size of the observed differences, random sampling error is not a plausible account for the effects. And even if you unreasonably assume that it might have been sampling error, that would still mean that any actual anti-black racism in hiring by these companies is so extremely rare that random sampling error could make the results come out in the extreme opposite direction.
Also, it isn't really a "single study". As those of us who conduct large complex research designs know, the difference between a "single" study and multiple studies is an arbitrary matter of how you choose to present it. They replicated the method in 3 regions and 3 different employment domains and analyze each separately. They could have just as easily presented them as multiple studies, and conceptually they are the same as replications (although by the same lab), each with enough observations to have the statically power to rule our sampling error. In no region or domain was their a bias in favor of white, and in all 3 regions and domains there was a numerical bias in favor of blacks, though it was too small to reach significance in 1 region and 1 domain.
Yes, unfortunately the domain of research on racial discrimination has suffered greatly from being dominated by ideologues with little regard for scientific rigor. They use methods that allow for many other factors to produce the result, then interpret it as only being due to race. Thus, there are almost no field studies in the last few decades that have manipulated perceived race directly and very little acknowledgment that this is a problem. I explicitly acknowledged the limitations of the study I referenced and didn't present it as evidence that their is no discrimination in other areas or more or less today. I present it as a contrast in method where the results can be more validly interpreted as a response to race itself, and that it's results do provide strong evidence that for a significant % of large companies there was more "reverse racism" than racism in 1980. Which does imply that there is likely reverse racism today, especially in contexts where the decision makers are likely to be motivated by either personal views, institutional policy, or fear of scrutiny to increase minority representation. It doesn't imply there is no anti-black discrimination, either in 1980 or today. But the meta-analysis you presented provides little valid evidence either way.
Regarding (a). First Lakisha was only one name used. In the first study cited, there were 36 names used altogether, including 9 different ones for 'African-American female'. Second, surnames were also included, so for example Lakisha was paired with the surname Washington, thereby adding to the racial clue and reducing the possibility of the person being mistaken for a foreigner. Finally, all names were matched against the most frequently used names, for the ethnic group they were intended to represent, in the region to which the resumes were sent.
The Lakisha-Emily paradigm is the most common and frequently adopted. Also, most the most common names for blacks are also common among whites, such that most people with that name are white. For example, there are more black boys named "Anthony" than "Tremanye" (one of the names used in the 36 name study), and yet most boys named "Anthony" in the US are white. Because whites outnumber blacks in the US by 6 to 1, names that are relatively common and has phonetic ties and roots to English words will imply a white person. The names that would reliably imply a non-white person are names that are overall rather uncommon and have foreign sounding roots. Which is why that study using 36 names used female names rooted in African and Arabic languages like Lakisha, Aisha, Kenya (and actual foreign country), while the "White" names were more like Emily, Jill, and, Sarah. And use of many names doesn't matter if the problem exists for some of the names. They don't show that the results emerge with each name, but rather lump the result together for the whole batch of white vs. black names. Just one or two names that imply to some employers a non-native English speaker would be sufficient to produce the reported results.
In addition, even if a name is highly common among blacks and only among blacks, that means that it is relatively uncommon to the most of those doing the hiring and would still be more likely to sound foreign to them. Plus, adding "Washington" to "Lakisha" only reduces the problem, it doesn't solve it. If a person is being thoroughly analytic about it, then sure, they should use all that information to consciously reach the conclusion that the applicant is likely an American black, but of course that is not how people work. The effect is subtle and usually below overt awareness. The name Lakisha still sounds like a foreign word and their brain reacts to it as such, with the accompanying "Washington" merely weakening that impact to be somewhere in between just "Lakisha" alone or just "Washington" alone. The effect only needs to be subtle, small, indirect, and on a small % of the employers to wind up accounting for the result.
Lots of very useful stuff there.
In general terms, my problem, if I were to try to respond individually or in detail to many of the caveats and criticisms (which are all valid, imo) would be that as an interested amateur and nothing more than that, I do not know how much
weight to give them in this discussion. In order to remedy this, I would in the the first instance, need to not only re-read the first study and the second meta-analysis, but also probably scrutinise each of the 21 studies the latter uses, and even then I'd arguably need to go further and look at other types of study, including the one you cited (which I can't find a free pdf for). I might even usefully investigate similar methodologies for studying things other than job recruitment. And even after all that I'd probably need to learn a lot more about the relevant sciences and their methodologies generally.
I am broadly fine with the suggestion that
'studies, especially in the social sciences, may not be finding what they say they are finding', because apart from anything else, human behaviour is possibly one of the most variable and capricious things that capricious humans could attempt to study. Whether I should go as far as to say that in this case, because of inherent limitations, the studies "provide little valid evidence either way", I am not sure,
and if I'm honest I'm still leaning towards thinking that they probably do show the things that they claim to, despite the limitations.
Here's the thing, their claims about racism in hiring can be true, and yet their experiments still not "show" that those claims are true. It is critical to keep these things separate. If the methodological problems are real, then by definition, the studies do not "show" what they claim, because that would require that there are not plausible (and in this case probable) alternative explanations. Do you think that some employers are less likely to give a call back to a non-native English speaking or foreign born applicant? Do you think that some employers would sometimes read an unfamiliar name rooted in a foreign language in infer (whether consciously or not) that the person might be non-native speaking or foreign born? If your answer to these questions is "yes" (and rationally it should be), then the logically neccessary conclusion is that the results observed in these studies would have emerged even in a world without anti-black discrimination. Data only "show" that a claim is true, if the data would only have plausibly emerged in a world where that claim is true. Thus, these data do not show the claim is true, despite the fact that the claim might be true anyway and that you might have other basis to believe it is true.
I think there is basis to think that there is racial discrimination in hiring, but I don't think there is a sound basis to believe that the prevalence of it has not declined at all in the last 30 years. In fact, there is indirect evidence that makes this claim implausible, such as evidence that racist beliefs have significantly declined during that period, that blacks have become more educated, with many more blacks holding positions of authority and respect, including on being elected and reelected president, which was unthinkable in the 1980's. Given that racist assumptions and feelings about blacks would be the mechanism behind hiring discrimination, this makes it implausible that hiring discrimination has not declined.
In contrast, what would be less likely to have declined over the years would be employer motives to not hire non-native English or foreign born applicants. Which makes this mechanism not only plausible, but a more plausible explanation for the lack of change in the size of the difference in callbacks observed in those studies over time.
You have other reasons, perhaps very good ones, to think that employers have a bias against black applicants. That doesn't mean that racial discrimination is the best explanation for these particular data, particularly the lack of decline in the observed result, which is the main reason you posted the meta-analysis to being with.
Other than things I have already offered, some more points in support of what I just said might be, for example, (a) that I read that the use of 'ethnic names' has supporting prior research to suggest that it is a useful indicator of the relevant attitudes,
That research merely shows that if you ask people to decide whether someone named "Lakisha" is more likely to be black or white, most people will say "black". But the baby registry data also show, that only 1 in a 1000 blacks are named Lakisha and only a small % named with other such race-signalling names. Which means the names signal being a special non-representative sub-set of that race. Plus, nothing in that prior research addresses that those names also signal other things only incidentally correlated with race, such as language skills, education quality, cultural background, etc.., and they are just much less familiar words and simple lack of familiarity produces subtle negative feeling like anxiety that could easily tip the scale in a some cases.
(b) that even if the 'ethnic names' resumes is not a perfect approach, it still has a lot going for it, not least the amount of control that can be exercised over all other aspects of the material used, (c) the 2 studies I have read most closely do seem to be pretty rigorous, such as in questioning their own limitations (although not the main one you have highlighted) and trying to correct for things like possible publication and write-up bias,
They are rather dismissive of the limitations they briefly mention, and then go right on making strong causal conclusions. And the limitations I'm bringing up would be rather obvious to any expert in the field, so their failure to deal seriously with them suggests a deliberate effort to avoid noting the more serious flaws with their methods.
(d) the results of 'in person' alternative type of studies, which of course may suffer from slightly different potential limitations, do seem to corroborate the 'blind resume' studies, (d) despite all that you say (and I have noted it) if we accept that 'there is racism' I still think it would be overall slightly odd if it was not going on in this sphere of human activity,
This relates directly to my first point above. You are using other information and data outside these studies to support the general conclusion of racism in hiring (which is fine and rational to do), but then using that to decide that these studies empirically show that this conclusion is true (which is not valid to do). What the studies empirically demonstrate and what you think is true due to other information are separate things. And note that even if you think racism would have to have some impact on hiring that doesn't support the conclusion of this meta-analysis that this impact has not changed in the last 30 years.
and finally, (e) even if the 'African-sounding' names do trigger a (possibly automatic or subconscious) response, and even if in some cases this prompts the reader to think 'possibly foreign', this may still at the very least indicate some sort of 'othering' bias
Sure, but it's a very different type of "othering", especially when it conjures the ideas that the person may have limited proficiency with the language essential for them to perform their job effectively.
and since the names are African-sounding and not for example Irish- or Russian-sounding, it may still be indicative of a racial bias in terms of a particular ethnicity.
Sure, this might be true and it might not be true. The studies do not show either way. They merely provide data consistent with that hypothesis but also consistent with several other hypotheses for which we also have other prior information to suggest are true. My ability to predict someone's gender from their name is consistent with me being psychic, but because it is also consistent with other highly plausible explanations, it does not show I am psychic or even qualify as evidence of it. And while you have more a priori reason to doubt that I am psychic that is a separate issue from when the data in question show.
Also, you just unwittingly hit upon a very simple method that would vastly improve these studies that none of these researchers has bothered to try all these decades: Simply make the "white" names foreign sounding names from predominantly white countries like Russia. What does it say about the competence of the researchers in this area that in 30 years, virtually none (and maybe truly none) have even bothered to mention this issue of the foreign/language implications of the "black" names, let alone bothered to improve their method by using "white" names with similar foreign/language implications?
All that said, if as you put it, "the domain of research on racial discrimination has suffered greatly from being dominated by ideologues" then all bets would be off. That, actually, is quite a damning claim, but I have no way of assessing it other than to say than I can see how it could be true. I would be curious to know if you would go as far as to suggest that that sort of bias (which I'm provisionally assuming you are suggesting is in favour of finding racism, rather than the opposite sort of bias) is why photos and/or explicitly-stated ethnicities are not part of such studies, for example. Or if not that, then why do those items (which, on the face of it, would seem likely, as you say, to improve the methodologies) not seem to be used more often nowadays?
Well, the early studies that did use more explicit race information (like the study I referenced) did not reliably show discrimination and sometimes showed reverse discrimination. Given that the methods researchers switched to as at least if not much more problematic in allowing causal inference, it is hard to imagine and honest intellectual justification why they would not at least give equal weight to both types of approaches and evaluate theories in light of all the data. Which leaves some kind of motivation to find a different result as a rather plausible account.
I obviously cannot know whether the poor quality of research and lack of serious consideration of the methodological problems is due to incompetence, deliberate dishonesty, and a mix of the two. Perhaps the use of racial names seemed honestly like a good idea at first and was just another way of approaching the problem, and when it gave more desired results than more direct manipulations of race, researchers came to preferred it.
What I do know is that this is just one example of this example of the lack of rigor and honest dealing with methodological problems in the field of racism research. The glaringly obvious problems with measures of "implicit racism" are another, and arguments about those problems almost always come from outside the field with racism researchers continuing to employ the methods and make unqualified conclusions about people racism from the data. Another is that almost all studies only use white participants. This may be b/c when non-white participants are also used they show the same pattern of results, and this undermines the conclusion that the observed response is motivated by racist beliefs rather than some other factor only incidentally associated with race.
Also, regarding dominating ideologies, my general impression is that there may be at least as many 'interested parties out there' who would surely be pleased to debunk studies showing persistent racism, and many of said interested parties would not lack the financial resources needed to have the necessary work carried out, and published widely if it concurred with their prior preferences or expectations.
There are right wing ideologues who occasionally put out papers to disprove racial discrimination. However, they don't do original empirical research, but rather are usually people with an economics background who take existing large datasets and conduct complex multivariate correlational analyses on them. They are often employed by right-wing "Think-tanks" like the Heritage Foundation.
But studies on racism that actually design and conduct empirical experiments happen almost entirely within academic social science departments like sociology, social psychology, and ethnic studies. And the research is done by people who chose to make a career out of studying racism, because as in all the sciences, people specialize and most do all their research within a rather narrow subfield. There are very very few conservatives in those departments at all, let alone those who chose to make a career of the sort who would want to manufacture evidence of non-discrimination. Mostly that is because right wingers generally dismiss the social sciences (and academia in general) as not being a valuable or respectable profession. This is partly b/c reality has a liberal bias, so conservatives tend to devalue science to seeks to reveal that reality. But also b/c people aren't going to make a career out of trying to debunk a particular hypothesis, partly b/c that's just not enough motivation and partly b/c it's nearly impossible to get research published in any area of social science when your data show null results (e.g., no effect of race on the outcome measure). Then, of course, there is the "ironic" problem of a person who does research that supports conservatives views having to be hired by people who are overwhelmingly liberal to extreme left and who are in favor of policies like AA that use race to increase minority representation.