• Welcome to the new Internet Infidels Discussion Board, formerly Talk Freethought.

The Crisis in Sociology

That seems to be a general problem with the social sciences: they seem short on broadly-applicable theories.
It's not a virtue to pretend that you have grand solutions or all-encompassing paradigms if you haven't. Sociology is yet a very young discipline in the grand scheme of things, and they produce plenty of new knowledge and data each year. I disagree that the turn against grand theory is a bad thing. Rather, it is a reflection of lessons hard learned during the very first few decades of the discipline. Of the social sciences in general. People were really and substantively hurt by some of the mistakes of Freudianism, Malthusianism, Marxism, and many other early sociological paradigms. Are still being hurt in some cases. We've learned to be much more careful about what we put into print when publishing on societal matters.
There seems to have been a pretty wide variety kf research methodoligies in the history of sociology, and some of them seem a lot more scientific than others.

For instance, how did Marx or Weber figure out their theories? Did they collect data systematically and test hylotheses? It seems like their approach was more like a kind of philosophy where they analysed an assortment of facts about society and came up with some interesting but untested (untestable?) ideas.

On the other hand, Durkheim seem to have been more systematic, collecting data on subjects methodically and making testable predictions.

Ricbard Feynmen criticised the social sciences for failing to put in the work required to know things, to establish facts and scientific theories.

Present-day sociologists have at least some of the tools they needs to do science: statistical methods and modern data analytics technologies come to mind. But I would imagine that, due to the nature of their subject matter, it is still difficult to collect data in the first place.
 
For instance, how did Marx or Weber figure out their theories? Did they collect data systematically and test hylotheses? It seems like their approach was more like a kind of philosophy where they analysed an assortment of facts about society and came up with some interesting but untested (untestable?) ideas
And yet, that sort of thing is what is being valorized here as "grand theory", and sociology excoriated for not producing more of. I don't think you are giving either Marx or Weber sufficient credit here - both did engage in scientific study, not just philosophy - but the cracks and biases affecting what they produced seem very obvious in retrospect.
 
For instance, how did Marx or Weber figure out their theories? Did they collect data systematically and test hylotheses? It seems like their approach was more like a kind of philosophy where they analysed an assortment of facts about society and came up with some interesting but untested (untestable?) ideas
And yet, that sort of thing is what is being valorized here as "grand theory", and sociology excoriated for not producing more of. I don't think you are giving either Marx or Weber sufficient credit here - both did engage in scientific study, not just philosophy - but the cracks and biases affecting what they produced seem very obvious in retrospect.

You make some good points. Maybe the field needs more physics/biology, but needs to be careful about broader application. Because obviously people, in practice, just take whatever's convenient to their interests and run with it.
 
Back to the PLoS paper.
Numerous studies have taken a direct approach, and have attempted to compare the hardness of two or more disciplines, usually psychology or sociology against one or more of the natural sciences. These studies used a variety of proxy measures including: ratio of theories to laws in introductory textbooks, number of colleagues acknowledged in papers, publication cost of interrupting academic career for one year, proportion of under 35 s who received above-average citations, concentration of citations in the literature, rate of pauses in lectures given to undergraduates, immediacy of citations, anticipation of one's work by colleagues, average age when receiving the Nobel prize, fraction of journals' space occupied by graphs (called Fractional Graph Area, or FGA), and others.
Fractional Graph Area? That's what one needs to for one's numbers. "One parameter, FGA, even appears to capture the relative hardness of sub-disciplines: in psychology, FGA is higher in journals rated as “harder” by psychologists, and also in journals specialised in animal behaviour rather than human behaviour."

But the idea of a hierarchy of sciences remains controversial, with some people claiming that scientific theories are nothing more than intellectual fashions. There is also some evidence of lack of hierarchy.
The consensus between scientists within a field, measured by several independent parameters including level of agreement in evaluating colleagues and research proposals, is similar in physics and sociology. The heterogeneity of effect sizes in meta-analyses also appears to be similar in the physical and the social sciences, suggesting a similar level of empirical cumulativeness. Historical reconstructions show that scientific controversies are common at the frontier of all fields, and the importance and validity of experiments is usually established in hindsight, after a controversy has settled.
Then some cases of previous statements influencing claimed results. "In evolutionary biology, published estimates on the heritability of sexually selected traits in various species were low for many years, but then suddenly increased when new mathematical models predicted that heritability should be high."

I was reminded of this:
The first X-ray-diffraction measurements were not much different from the first measurements, Millikan's oil-drop experiments, but they soon settled down to the present value.
 
For instance, how did Marx or Weber figure out their theories? Did they collect data systematically and test hylotheses? It seems like their approach was more like a kind of philosophy where they analysed an assortment of facts about society and came up with some interesting but untested (untestable?) ideas
And yet, that sort of thing is what is being valorized here as "grand theory", and sociology excoriated for not producing more of. I don't think you are giving either Marx or Weber sufficient credit here - both did engage in scientific study, not just philosophy - but the cracks and biases affecting what they produced seem very obvious in retrospect.
I'm pretty ignorant about the field and it's history, so naturally my view is too simplistic. I will have to read up a bit more about their methods.
 
The contrast between indirect measures of hardness, which point to a hierarchy, and evidence of high controversy and disagreement in all kinds of research has inspired an intermediate position, which distinguishes between the “core” and the “frontier” of research. The core is the corpus of agreed upon theories and concepts that researchers need to know in order to contribute to the field. Identifiable with the content of advanced university textbooks, the core is clearly more developed and structured in the physical than in the social sciences. The frontier is where research is actually done, where scientists produce new data and concepts, most of which will eventually be contradicted or forgotten and will never make it to the core. At the frontier, levels of uncertainty and disagreement might be similar across fields.

What did author Daniele Fanelli find? A lot of overlap over categories, but pure soft sciences had more positive results than pure hard ones, while applied ones were about the same, and similar to the pure soft ones.
Overall, however, they support the existence of a Hierarchy of the Sciences, in which scientific rigour and objectivity are roughly inversely proportional to the complexity of subject matter and possibly other field-specific characteristics (e.g. level of development, see below). On the other hand, the differences observed were only a matter of degree. This supports the scientific status of the social sciences against claims that they are qualitatively different from the natural sciences and that a scientific method based on objectivity cannot be applied to them.
So the social sciences are "real" sciences.
Not all observations matched the predicted hierarchy, however. At the disciplinary level, Psychology and Psychiatry had more positive results than Social Sciences, General, contradicting previous studies that placed psychology between biology and sociology. Moreover, Physics and Chemistry had more positive results than Social Sciences, General and a few biological disciplines. At the level of methodology, biological, non-behavioural studies on humans and non-humans had more positive results than behavioural studies on non-humans. At both levels, papers in applied disciplines showed a markedly different pattern, having uniformly high frequencies of positive results.
 
It's interesting to me to see this blistering critique of sociology, incidentally, owing to my own profession and the way it was taught to me. To be frank, cultural anthropology and sociology have a common history in some ways, but also a very rough history ever since they came to be seen as separate fields. In short, the attitude many of my professors held toward sociology was dismissive at best, and the sentiment was returned with gusto. All through my student years, there was this implicit awareness that both fields turned their noses up at one another a little bit, very specifically due to a supposed lack of scientific rigor. Sociologists regarded their very quantitative approach, based heavily on statistical analysis of survey data, as scientifically superior to the overly freewheeling, qualitative approach of cultural anthropologists (as they saw it). We were compromised, they would argue, by our love of ethnographic study and the inevitable bias that comes from studying in situ. While cultural anthropologists commonly regarded the narrower focus of sociology as nothing less than unjustifiably tossing out relevant data. If sociology couldn't work as a field science, was it a science at all? Surveys put too much power in the hands of researchers, alleged anthropologists, to define what kinds of data their studies could produce. The common lack of any reference to the biological elements of life were also a common avenue of critique, that by cross-training in physical anthropology, cultural anthropologists were much better prepared to comment on those matters where the physical and the social overlap.

In professional life, I have found these critiques less than useful. There's some justification for both portrayals, but they can also easily become inaccurate caricatures in the hands of over-excited grad students or aging boomers worried about their theoretical legacies. In real life, sociologists experiment with ethnography all the time. Anthropologists use survey. Everyone has opportunities to incorporate biological insights into their work if interested, and plenty of people in both fields have done so over the years. As time goes by, my alarm at the overall threat faced by the social sciences as a whole has come to be a much more serious preoccupation of mine. I cannot imagine the value of defining and rhetoricizing the presumed differences in the theoretical sets of sociology vs anthropology, sociology vs economics, psychology vs anthropology, and so forth, when all of these disciplines are under serious threat of dissapearing altogether. If sociology is somehow able to weather the storm and survive as anthropology and political science go under or get absorbed, at this point in my life I'd be perfectly content to change the job title on my desk as long as the knowledge generated by these fields is not allowed to die. We are at a precarious stage, and theoretical concerns are not the reason why.
 
I don't know whether sociology suffers from this, but my experience of reading psychology papers has left me with the impression that psychology is almost exclusively the study of the behaviour of the subset of middle class American psychology sophomores who are prepared to participate in studies in return for small incentives.

Surveys and questionnaires are inevitably of limited value when the respondents are the people who happen to be available at a university campus, who are likely to be unrepresentative of the wider public in any number of ways.
 
DF states: "Hypotheses tested in biological and social sciences could have a higher probability of being true."

Then asks why that might be.


Truth value?

Prior knowledge and beliefs?
"Scientists in softer sciences might chose their hypotheses based on a greater amount of personal observations, preliminary results, and pure and simple intuition that precede a “formal”, published “test” of the hypothesis."

Depth of hypotheses tested?
Younger, less developed fields of research should tend to produce and test hypotheses about observable relationships between variables (“phenomenological” theories). The more a field develops and “matures”, the more it tends to develop and test hypotheses about non-observable phenomena underlying the observed relationships (“mechanistic” theories). These latter kinds of hypotheses reach deeper levels of reality, are logically stronger, less likely to be true, and are more conclusively testable.

Rigor of hypothesis testing?

Flexibility in doing research?
In sciences that are younger and/or address phenomena of higher complexity, the connection between theories, hypotheses and empirical findings could be more flexible, negotiable and open to interpretation. This would give scientists more freedom in deciding how to collect, analyze and interpret data, which increases the chances that they will produce a support of the hypotheses they believe to be true. In its earliest stages of development, a discipline or field can be completely fragmented theoretically and methodologically, and have different schools of thought that interpret the same phenomena in radically different ways –a condition that seems to characterize many fields in the social sciences and possibly some of the biological sciences.

Experimenter effects?
 Observer-expectancy effect
The biasing effect of researchers' expectations is increasingly recognized in all disciplines including physics, but has been most extensively documented in the behavioural sciences.

Non-publication of negative and/or statistically insignificant results?
The file-drawer effect:  Publication bias

"Prevalence and strength of manipulation of data and results"?
Outright fakery?
Several factors are hypothesised to increase scientists' propensity to falsify research, including: the likelihood of being caught, consequences of being caught, the costs of producing data compared to publishing them, strong belief in one's preferred theories, financial interests, etc…

Survey data suggests that outright scientific misconduct is relatively rare compared to more subtle forms of bias, although it is probably higher than commonly assumed, particularly in medical/clinical research.
 
More Daniele Fanelli:
Papers testing multiple hypotheses were more likely to report a negative support for the first hypothesis they presented. This suggests that the order in which scientists list their hypotheses follows a rhetorical pattern, in which the first hypothesis presented is falsified in favour of a subsequent one.

And this bit:
Given what sociologists have sometimes written about sociology (e.g. that it is probably the only science where knowledge is truly socially constructed), economists of economics (e.g. that econometrics is like alchemy, with regression analysis being it's philosopher's stone), and psychiatrists of psychology and psychiatry (e.g. that they “pretend to be sciences, offering allegedly empirical observations about the functions and malfunctions of the human mind”), it could be surprising to find any negative results at all in these disciplines. As argued above, this study suggests that such categorical criticisms of the social sciences are excessive.
 
I don't know whether sociology suffers from this, but my experience of reading psychology papers has left me with the impression that psychology is almost exclusively the study of the behaviour of the subset of middle class American psychology sophomores who are prepared to participate in studies in return for small incentives.
WEIRDS, as a classic critical article once put it: Western, Educated, Industrialised, Rich, Democratic.
 
I don't know whether sociology suffers from this, but my experience of reading psychology papers has left me with the impression that psychology is almost exclusively the study of the behaviour of the subset of middle class American psychology sophomores who are prepared to participate in studies in return for small incentives.
Yes, WEIRD people - "Western, Educated, Industrialized, Rich, and Democratic"
 
There seems to have been a pretty wide variety kf research methodoligies in the history of sociology, and some of them seem a lot more scientific than others.

For instance, how did Marx or Weber figure out their theories? Did they collect data systematically and test hylotheses? It seems like their approach was more like a kind of philosophy where they analysed an assortment of facts about society and came up with some interesting but untested (untestable?) ideas.

On the other hand, Durkheim seem to have been more systematic, collecting data on subjects methodically and making testable predictions.
Over much of the natural sciences, it is difficult or impossible to experiment on the subject matter, yet those fields are still considered very good science and very successful science.

Consider Earth and space sciences. I group them together because drawing a sharp line between them is about as sensible as drawing a sharp line between physics and chemistry. There is not much experimenting that one can do, mainly on upper-crust rocks, soil, bodies of water, the air, meteorites, and the bits of material that have been brought back from the Moon and some asteroids. One can go a bit further and make earthquakes by big explosions and follow spacecraft radio broadcasts as they go through atmospheres, but beyond that, it's just about all observation.

Yet Earth and space sciences are considered well on the hard side, and very successful at that.

Turning to some softer sciences, the biological sciences, much of those are essentially observational, especially paleontology and evolutionary biology. Yet evolutionary biology is very successful -- we have worked out a family tree of *every* well-studied cellular organism.

So it *is* possible to test hypotheses without doing experiments.
 
It's interesting to me to see this blistering critique of sociology, incidentally, owing to my own profession and the way it was taught to me. To be frank, cultural anthropology and sociology have a common history in some ways, but also a very rough history ever since they came to be seen as separate fields. In short, the attitude many of my professors held toward sociology was dismissive at best, and the sentiment was returned with gusto. All through my student years, there was this implicit awareness that both fields turned their noses up at one another a little bit, very specifically due to a supposed lack of scientific rigor. Sociologists regarded their very quantitative approach, based heavily on statistical analysis of survey data, as scientifically superior to the overly freewheeling, qualitative approach of cultural anthropologists (as they saw it). We were compromised, they would argue, by our love of ethnographic study and the inevitable bias that comes from studying in situ. While cultural anthropologists commonly regarded the narrower focus of sociology as nothing less than unjustifiably tossing out relevant data. If sociology couldn't work as a field science, was it a science at all? Surveys put too much power in the hands of researchers, alleged anthropologists, to define what kinds of data their studies could produce. The common lack of any reference to the biological elements of life were also a common avenue of critique, that by cross-training in physical anthropology, cultural anthropologists were much better prepared to comment on those matters where the physical and the social overlap.

In professional life, I have found these critiques less than useful. There's some justification for both portrayals, but they can also easily become inaccurate caricatures in the hands of over-excited grad students or aging boomers worried about their theoretical legacies. In real life, sociologists experiment with ethnography all the time. Anthropologists use survey. Everyone has opportunities to incorporate biological insights into their work if interested, and plenty of people in both fields have done so over the years. As time goes by, my alarm at the overall threat faced by the social sciences as a whole has come to be a much more serious preoccupation of mine. I cannot imagine the value of defining and rhetoricizing the presumed differences in the theoretical sets of sociology vs anthropology, sociology vs economics, psychology vs anthropology, and so forth, when all of these disciplines are under serious threat of dissapearing altogether. If sociology is somehow able to weather the storm and survive as anthropology and political science go under or get absorbed, at this point in my life I'd be perfectly content to change the job title on my desk as long as the knowledge generated by these fields is not allowed to die. We are at a precarious stage, and theoretical concerns are not the reason why.

I did notice that the book in my post was published in 2001, I imagine it was a very different world back then.

I skimmed it briefly a few weeks ago, and what I picked up from it wasn't so much about rigor per se, but rather subsets of the discipline completely rejecting scientific fact, like gender differences and the like. How many in the field have kind of signed up for the Blank Slate, rather than internalizing what we actually know from the study of biology.

The science being done, in most cases, likely at least approaches rigorous. But your analysis is going to be biased if you reject (or don't understand) scientific axioms.

I saw a pretty good example of this on Mastodon recently, where a study found that social media kept people from 'transformative boredom'. A totally plausible and convincing conclusion, but it didn't really explain what was actually happening. More accurately, people prioritize social media over other activities because, in the short term (how biology works) connecting with other people, forming friendships, eventually sleeping with somebody is more important. But absolutely nobody would have learned that from the study, they'd just get the idea that people should be throwing their phones in the lake so they can take up oil painting.

The book also went on for a bit about how the field never truly materialized into a coherent body of knowledge, but I didn't look as closely at that part.
 
Last edited:
WEIRDS, as a classic critical article once put it: Western, Educated, Industrialised, Rich, Democratic.
The thing that struck me was their youth.

I was looking at studies of driver behaviour and attitudes, and they all show that very large numbers of people are taking a lot of risks, and are unaware of the magnitude of the risks they're taking.

But as the participants are all college students, presumably the sample is almost exclusively drawn from the under-25 demographic, that any insurer can tell you are vastly more commonly involved in crashes than over-25s.

And presumably the strategies that are being implemented in an attempt to improve driver behaviour are potentially completely inappropriate for more experienced drivers - though if they are, we will probably never know because only Californian college kids are being looked at at all.

The assumption that the behaviour of an eighteen year old Californian can advise strategies to improve driving habits for fifty year old Queenslanders seems pretty dubious to me - but as our politicians and their advisers only read the executive summary of the abstract of these papers, they have no idea that that's what is happening.
 
WEIRDS, as a classic critical article once put it: Western, Educated, Industrialised, Rich, Democratic.
The thing that struck me was their youth.

I was looking at studies of driver behaviour and attitudes, and they all show that very large numbers of people are taking a lot of risks, and are unaware of the magnitude of the risks they're taking.

But as the participants are all college students, presumably the sample is almost exclusively drawn from the under-25 demographic, that any insurer can tell you are vastly more commonly involved in crashes than over-25s.

And presumably the strategies that are being implemented in an attempt to improve driver behaviour are potentially completely inappropriate for more experienced drivers - though if they are, we will probably never know because only Californian college kids are being looked at at all.

The assumption that the behaviour of an eighteen year old Californian can advise strategies to improve driving habits for fifty year old Queenslanders seems pretty dubious to me - but as our politicians and their advisers only read the executive summary of the abstract of these papers, they have no idea that that's what is happening.
I will say this, that psychologists in general are aware of this problem and actively trying to mitigate and control for it in various ways. But recruiting from the general public for psychological studies is expensive and difficult. Potential grantors love being told that you need triple the money for your project because college students are a non-representative sample of the global population!
 
WEIRDS, as a classic critical article once put it: Western, Educated, Industrialised, Rich, Democratic.
The thing that struck me was their youth.

I was looking at studies of driver behaviour and attitudes, and they all show that very large numbers of people are taking a lot of risks, and are unaware of the magnitude of the risks they're taking.

But as the participants are all college students, presumably the sample is almost exclusively drawn from the under-25 demographic, that any insurer can tell you are vastly more commonly involved in crashes than over-25s.

And presumably the strategies that are being implemented in an attempt to improve driver behaviour are potentially completely inappropriate for more experienced drivers - though if they are, we will probably never know because only Californian college kids are being looked at at all.

The assumption that the behaviour of an eighteen year old Californian can advise strategies to improve driving habits for fifty year old Queenslanders seems pretty dubious to me - but as our politicians and their advisers only read the executive summary of the abstract of these papers, they have no idea that that's what is happening.
I will say this, that psychologists in general are aware of this problem and actively trying to mitigate and control for it in various ways. But recruiting from the general public for psychological studies is expensive and difficult. Potential grantors love being told that you need triple the money for your project because college students are a non-representative sample of the global population!
This attitude is why the proposed LHC at CERN was replaced with a sophomore banging two housebricks together. It's not as informative about reality as colliding protons at a substantial fraction of lightspeed, but it's a LOT cheaper and easier.

It seems psychologists need to look to the harder sciences for advice on funding, rather than on mere disciplinary rigour. :)
 
Back
Top Bottom