Critics of sociology stating that the discipline has no ‘excess empirical content’, ignore efforts of methodological rigor without which ‘anything goes’

It is sometimes said that sociology is particularly well fit to kick in open doors. It is a response to sociological research findings is that people think they already knew the outcome of this study. Sure, sometimes this may be the case, but most often one may have a suspicion about the outcomes – rather than actual knowledge.

In philosophical sense, this criticism boils down to arguin that most sociological theories have no excess content: they have nothing new to add to what we already knew. One argued that is often heard is that in sociology so many methods are used that one can always find a method – or alter one – to support one’s own ideas: sociology would lack methodological rigor.

The term ‘excess empirical content’ was coined by Imre Lakatos. In my dissertation I write about my philosophy of science:

Based on Lakatosian philisophy of science, the interaction between opportunities and interests means that we must reject explanations of women’s employment solely based on the concept of opportunities. Lakatos argued that: “For the sophisticated falsificationist a scientific the- ory T is falsified if and only if another theory T’ has been proposed with the following characteristics: (1) T’ has excess empirical content over T: that is, it predicts novel facts, that is, facts improbable in the light of, or even forbidden, by T; (2) T’ explains the previous success of T, that is, all the unrefuted content of T is included (within the limits of observational error) in the content of T’; and (3) some of the excess content of T’ is corroborated.” (Lakatos, 1978, p. also see: Levels & Nieuwenhuis, 2011).

Paul Feyerabend argued for methodological pluralism, stating that ‘anything goes’. This has been interpreted as indicating that any method should be endorsed, or tolerated. I do not think this is a correct interpretation. I interpret Feyerabend as suggesting that one should look at problems in different ways – and see whether news ideas reject old conceptions of truth. The methodological rigor is then found in the attempt to relate the findings based on different methodologies to each other – within a common theoretical framework.

In my dissertation I have closed three open doors, so to say, or rejected (at least) three theories:

• Pettit and Hook (2009) argued that (parental) leave is a mechanism of exclusion of women from the labor market. In my analyses I found that short-term leave includes women on the labor market, and only very long periods of leave exclude women from the labor market
• As argued in the quote above, I reject explanations of the outcomes of family policies solely based on opportunities, as it is the interaction of opportunities and women’s interest in employment that drive the outcomes of family policies.
• The incomplete revolution thesis by Esping-Andersen, who argued that increasing rates of women’s employment resulted in higher degrees of inequality between households. I could not confirm this claim, and rather argued that “The conditions for women’s earnings to increase inequality between households are hard to meet“.

——
This is a series on the 10 propositions that are part of my PhD dissertation. These propositions are a Dutch tradition to highlight key findings of a dissertation and some additional insights by the author. My dissertation is titled “Family Policy Outcomes: Combining Institutional and Demographic Explanations of Women’s Employment and Earnings Inequality in OECD countries, 1975-2005″ and I will defend my dissertation on January 10 2014. So, this series is also a count down. Find out more about my dissertation.

Can bad science lead us to draw wrong conclusions about the world we live in? “Of course it can“, we are inclined to think. And if so, can this have real-life consequences? Investigating these meta-questions is not as easy as it might seem, for it would require an exact manner to distinguish the good from the bad science, and it would require a subject that has been thoroughly investigated in both the ‘good’ and the ‘bad’ ways to compare the outcomes.

One such subject would be the vast amount of research done on the psychological consequences of undergoing an induced abortion. This heavily researched (and heavily debated!) subject focusses primarily on the questions whether or not a women has a higher chance of suffering from anxiety, feeling of guilt, depression, or (other) mental disorders caused by undergoing an induced abortion. The conclusions drawn in the vast literature on this basic question vary form an abortion having no consequences, to an abortion having a negative impact on the psychological well-being of a woman.

Charles, Polis, Sridhara and Blum (2008) did a systematic review of this literature. Interestingly, they did not only classify the findings, but also derived a set of guidelines by which to evaluate the methodological quality of each research paper.

Analytical studies were rated on a scale from Excellent to Very Poor using five major criteria: (1) use of an appropriate comparison group; (2) use of valid mental health measures; (3) control for preexisting mental health status; (4) control for confounders; and
(5) whether there was comprehensive exploration of the research question. (p. 437)

Trained researchers would find it difficult to disagree with these ‘five guidelines for quality research’. However, hardly any of the studies they evaluated did comply with all of them. Try to imagine what the consequences of failing to comply with any of these guidelines might be. For instance, it is known that mental health status relates to the chance of becoming pregnant unintendedly, and subsequently it relates to the decision whether or not to keep the baby. Not taking that into account would attribute the difference in mental health between women who have had an abortion, and women who did not, to the abortion, whereas these differences in fact were already present before the abortion. A similar argument goes for taking into account differences regarding age, educational level, and religious conviction: these characteristics all relate to both mental health and the odds of having an unintended pregnancy and / or an abortion. As a last example, the groups of women that are compared should be as equivalent as possible, except for having had an abortion. However, studies compared women having had an abortion with, for example, the general population, or just with women having carried out their pregnancy. This practice does not allow for causal inference, for it cannot exclude the effects of pregnancy intentions.

In total, Charles et al. evaluated 21 studies, the methodological quality of which varied widely, as did their conclusions on the consequences of an abortion. Their most fascinating finding was that a relation exists between the methodological quality of a study and the conclusions drawn in it: the better studies hardly found any consequences from undergoing an abortion, whereas the poorer quality studies did find negative consequences of an abortion. In other words: bad science tends to overestimate the consequences of an induced abortion.
Generally, these differences are due to the fact that the studies with poor methodology did not take into account differences between women prior to the abortion, thereby attributing differences between women in mental health to having had an abortion, whereas in fact these differences were pre-existing.

There we have it: bad science distinguished from good science, and different outcomes. So, yes, bad science indeed leads to wrong outcomes and in real life, this can have consequences. In the policy-discussions on induced abortion, the possibility of negative consequences of an abortion plays an important role. It would depend on which research papers the policy makers turn to, what their stand on the issue would be.

The question that remains is now: can we trust policy makers to distinguish the good from the bad (science)?

V CHARLES, C POLIS, S SRIDHARA, R BLUM (2008). Abortion and long-term mental health outcomes: a systematic review of the evidence Contraception, 78 (6), 436-450 DOI: 10.1016/j.contraception.2008.07.005