The Dunning-Kruger Effect Debunked

[jab]

Here's an interesting article I came across recently. In a nutshell, it explains that a statistical error was made in Dunning and Kruger's original paper, and critiques have since debunked the effect. But the original paper is far more widely cited, so a lot of people believe the effect is real.

The Dunning-Kruger Effect is Autocorrelation – Economics from the Top Down

Do unskilled people actually underestimate their incompetence?

economicsfromthetopdown.com

It's not a statistical error. It's a methodological error. And it's not really an error (more presentation peculiarity), conclusion is still valid. Dumb people think that they are smarter than they actually are and smart people unaware how dumb the rest of the people are.
Original plot is weirdly organized but it's not correlation plot, it does not have to comply with this auto-correlation thing.
Dunning and Kruger are psychologists, not physicists or even mathematicians.
And effect itself is kinda obvious to say the least.

re ovbiousness: It is kind of obvious that the earth is flat.

 

[JohnG]

I know it's just a proverb, but I think there is truth in the saying "The more you know the more you realize you don't know" (or something like that),

Along the lines of "A little knowledge is a dangerous thing"

You see examples of this on a daily basis and we can all be guilty of it from time to time

 

[TomC]

I know it's just a proverb, but I think there is truth in the saying "The more you know the more you realize you don't know" (or something like that),

Along the lines of "A little knowledge is a dangerous thing"

You see examples of this on a daily basis and we can all be guilty of it from time to time

Yeah...
This.

Humans are consistently stupid, self centered, and habit prone.
Whether it's because we evolved that way or because God made us that way, it's human nature. We're inclined to behave badly then justify it later, if we're asked.
(Or charged)
Tom

 

[JohnG]

I know it's just a proverb, but I think there is truth in the saying "The more you know the more you realize you don't know" (or something like that),

Along the lines of "A little knowledge is a dangerous thing"

You see examples of this on a daily basis and we can all be guilty of it from time to time

Yeah...
This.

Humans are consistently stupid, self centered, and habit prone.
Whether it's because we evolved that way or because God made us that way, it's human nature. We're inclined to behave badly then justify it later, if we're asked.
(Or charged)
Tom

It's not always the case, but quite often the most confident people I know are also the dumbest.

 

[TomC]

It's not always the case, but quite often the most confident people I know are also the dumbest.

If there is a God,

Either He keeps making us that way, generation after generation, because He's a bumbler or because He's profoundly immoral.

I'm inclined to go with He doesn't make us any way. We just are, there is no God who even notices, much less cares.
Tom

 

[Don2 (Don1 Revised)]

I started reading the blog linked in the op.

I am not completely convinced by their argument. I actually do think auto-correlation is a little bit of an issue, but not as much as they make it out. More on that...

Essentially, they are saying, "Look, if we throw random numbers in, we can replicate D-K."

So, first, the point of observation of the D-K effect is that persons who are in the lower bracket(s) are very unreliable about their own knowledge. The opposite of D-K, i.e. trying to disprove D-K, would entail not simply choosing random numbers for x and y, but instead having the lower bracket choose random error within some reasonable boundary about x vs x. Then, see what it looks like. If you are choosing random numbers for x and y, then essentially, you are building in SOME of the stupidity of the population. Then, you are saying, "look, when the population is stupid, it looks like a D-K graph. So, the D-K graph must be wrong!!11" which is just a half-contradictory claim.

Second, the simulated graph doesn't quite look like the D-K graph. This is because the error between actual and predicted is symmetric in their result which they claim replicates D-K but it does not quite do so, i.e. they present the same magnitude error whether it's for the lower brackets or upper brackets:

But the D-K graph has a much larger error in the lower bracket and a smaller error in the upper bracket.

D-K observation was that there was much more positive error in lower brackets and smaller negative error in the upper bracket:

Which brings me to the third point of interest. Take a look at user comment#3 in the blog:

The conversion to percentiles introduces a second bias (in addition to the problem of autocorrelation). By definition, percentiles have a floor (0) and a ceiling (100), and are uniformly distributed between these bounds. If you are close the floor, it is impossible for you to underestimate your rank. Therefore, the ‘unskilled’ will appear overconfident. And if you are close to the ceiling, you cannot overestimate your rank. Therefore, the ‘skilled’ will appear too modest. See Nuhfer et al (2016) for more details.

I think this comment needs some tweaking. Suppose there is some unreliability or error about the true line so that a self-assessment is like <= +/- error, say, +/- 10, for example. When an individual has 100%, they can't overestimate their score. When an individual has 99%, they can only overestimate by 1% but there is far more room below, to go down to 89%. Consider the flip side of this observation as well. When an individual is 0%, they can't underestimate. When they are at 1%, there is a lot of room above to overestimate their ability. I am unsure that this creates a "bias" or that "the 'skilled' will appear too modest," as opposed to an alternate hypothesis that actual individuals WILL BE too modest.

In other words, we can talk about this as some kind of theoretical methodological flaw. But on the other hand, when you consider the real, natural world and social phenomena, there really is a cutoff of overestimation in elite individuals and a cutoff of underestimation in sub-par individuals.

In the world of pure math, it's a numerical artifact, but in the natural world, it may be a valid series of observations.

 

[Don2 (Don1 Revised)]

Here are some additional links of interest.

A good criticism of the op post's referenced blog article:

I can’t let go of “The Dunning-Kruger Effect is Autocorrelation”

Organizing my thoughts regarding an argument about the Dunning-Kruger study and statistics in general

andersource.dev

A very professional statistical paper regarding D-K:

A Statistical Explanation of the Dunning–Kruger Effect

An explanation of the Dunning–Kruger effect is provided which does not require any psychological explanation, because it is derived as a statistical artifact. This is achieved by specifying a simple statistical model which explicitly takes the (random) boundary constraints into account. The...

www.frontiersin.org

Interesting response from Dunning referencing criticisms and empirical support:

The Dunning-Kruger effect and its discontents | BPS

www.bps.org.uk

 

[Loren Pechtel]

I think this comment needs some tweaking. Suppose there is some unreliability or error about the true line so that a self-assessment is like <= +/- error, say, +/- 10, for example. When an individual has 100%, they can't overestimate their score. When an individual has 99%, they can only overestimate by 1% but there is far more room below, to go down to 89%. Consider the flip side of this observation as well. When an individual is 0%, they can't underestimate. When they are at 1%, there is a lot of room above to overestimate their ability. I am unsure that this creates a "bias" or that "the 'skilled' will appear too modest," as opposed to an alternate hypothesis that actual individuals WILL BE too modest.

In other words, we can talk about this as some kind of theoretical methodological flaw. But on the other hand, when you consider the real, natural world and social phenomena, there really is a cutoff of overestimation in elite individuals and a cutoff of underestimation in sub-par individuals.

In the world of pure math, it's a numerical artifact, but in the natural world, it may be a valid series of observations.

Another thought:

The less skilled are also less skilled at estimating their ability--combine that with the bounding and you would expect a greater deviation on the left than the right.

 

[Don2 (Don1 Revised)]

I think this comment needs some tweaking. Suppose there is some unreliability or error about the true line so that a self-assessment is like <= +/- error, say, +/- 10, for example. When an individual has 100%, they can't overestimate their score. When an individual has 99%, they can only overestimate by 1% but there is far more room below, to go down to 89%. Consider the flip side of this observation as well. When an individual is 0%, they can't underestimate. When they are at 1%, there is a lot of room above to overestimate their ability. I am unsure that this creates a "bias" or that "the 'skilled' will appear too modest," as opposed to an alternate hypothesis that actual individuals WILL BE too modest.

In other words, we can talk about this as some kind of theoretical methodological flaw. But on the other hand, when you consider the real, natural world and social phenomena, there really is a cutoff of overestimation in elite individuals and a cutoff of underestimation in sub-par individuals.

In the world of pure math, it's a numerical artifact, but in the natural world, it may be a valid series of observations.

Another thought:

The less skilled are also less skilled at estimating their ability--combine that with the bounding and you would expect a greater deviation on the left than the right.

It's interesting. If you look at the statistical paper in the link#2, I gave, they also claim to be able to replicate D-K, but they also do not. They do prove that an S shaped curve can be developed because of the artificial bounds. However, their lower bracket is seriously much lower than the lower bracket in the D-K graph. That said, I do think that the artificial bounds does have some kind of partial effect, just like you are thinking.

Where I differ is that if you look at D-K graph, the estimate given by lower bracket is greater than 50%. It's like there is some contingent in the lower bracket in denial that they are lower bracket. Imagine if you ask them how they did and they respond, "above average." A lot of people think they are above average. It isn't clear to me that you can model this with people giving random answers. Like the graph in the auto-correlation, if it is random answers, you almost get a flat line at 50%.

It isn't clear to me what the true answer is....but I think the critics do not provide solutions that explain everything and perhaps also D-K did not pay enough attention to statistical artefacts, too.

 

[Loren Pechtel]

I think this comment needs some tweaking. Suppose there is some unreliability or error about the true line so that a self-assessment is like <= +/- error, say, +/- 10, for example. When an individual has 100%, they can't overestimate their score. When an individual has 99%, they can only overestimate by 1% but there is far more room below, to go down to 89%. Consider the flip side of this observation as well. When an individual is 0%, they can't underestimate. When they are at 1%, there is a lot of room above to overestimate their ability. I am unsure that this creates a "bias" or that "the 'skilled' will appear too modest," as opposed to an alternate hypothesis that actual individuals WILL BE too modest.

In other words, we can talk about this as some kind of theoretical methodological flaw. But on the other hand, when you consider the real, natural world and social phenomena, there really is a cutoff of overestimation in elite individuals and a cutoff of underestimation in sub-par individuals.

In the world of pure math, it's a numerical artifact, but in the natural world, it may be a valid series of observations.

Another thought:

The less skilled are also less skilled at estimating their ability--combine that with the bounding and you would expect a greater deviation on the left than the right.

It's interesting. If you look at the statistical paper in the link#2, I gave, they also claim to be able to replicate D-K, but they also do not. They do prove that an S shaped curve can be developed because of the artificial bounds. However, their lower bracket is seriously much lower than the lower bracket in the D-K graph. That said, I do think that the artificial bounds does have some kind of partial effect, just like you are thinking.

Where I differ is that if you look at D-K graph, the estimate given by lower bracket is greater than 50%. It's like there is some contingent in the lower bracket in denial that they are lower bracket. Imagine if you ask them how they did and they respond, "above average." A lot of people think they are above average. It isn't clear to me that you can model this with people giving random answers. Like the graph in the auto-correlation, if it is random answers, you almost get a flat line at 50%.

It isn't clear to me what the true answer is....but I think the critics do not provide solutions that explain everything and perhaps also D-K did not pay enough attention to statistical artefacts, too.

I completely do not accept the autocorrelation "answer" for the reasons you state--there's something to it. How much of it is real vs an artifact of the bounding I do not know. It would be hard for there to not be some actual error in this regard because reality is divided up into things we know/things we know we don't know/things we aren't aware of to know we don't know about. If you don't know you don't know you won't count it--the DK people are inherently going to underestimate the size of the domain and thus think they know a bigger part of it than they actually do. On the high end you will have people who overestimate because they don't realize that it's not just them that doesn't know X, nobody does.

 

[Politesse]

Everybody thinks themselves smart until they get asked, "African or European swallow?"

Zackly!!!

What about the Welcome Swallow (Australia)?
https://birdlife.org.au/bird-profiles/welcome-swallow/

Have y'all any idea what various types of swallowing might mean to a gay guy who doesn't much care about birds?

Asking for a friend...
Tom

ETA ~Ha ha ha...
The "Welcoming Swallow? ~

Huh. We mostly have tree
swallows and barn swallows in our area. I guess that checks out. Watch out for those rough-wing swallows, though.

 

[Politesse]

As for the OP, I'm not sure the Dunning Kruger effect had that much of a reputation in the first place. Even if accurate, it seems diagnostically useless to social scientists or social workers, so why would anyone really care whether or not it were "debunked"? It is important in forensic contexts to understand that self report of skill is not a reliable metric, but would apply equally to all witnesses.

 

[T.G.G. Moogly]

It isn't clear to me what the true answer is....but I think the critics do not provide solutions that explain everything and perhaps also D-K did not pay enough attention to statistical artefacts, too.

As you've inferred, what would be wrong with plotting the data case by case, individual by individual instead of in groups? Granted there will be a parabolic tail but the gist of the graph ought to demonstrate whether the effect is real or not and could be easily replicated.

The problem, of course, is the same as in any psychological situation, and that is that people will sense something is up and react accordingly. Ideally you want to gather the information without anyone knowing they are being observed.

 

[Wiploc]

This article seemed on point to me when I first read it (hence posting), and I think there was a problem in the original study. But it looks like the ensuing academic conversation on Dunning-Kruger is a bit of a wormhole with a lot of facets to it.

Surely a Nobel Prize is due to the discoverer of the faceted wormhole!

 

[Cheerful Charlie]

I think that DK is not the only problem. There is the Willful Ignorance Problem that we see all too often.

 

[bilby]

I think that DK is not the only problem. There is the Willful Ignorance Problem that we see all too often.

I don't. I refuse to.

 

[T.G.G. Moogly]

Flat-earthers are overconfident about their own scientific knowledge but exhibit low scientific literacy, study finds

Furthermore, the researchers found that flat earth believers exhibited higher levels of overconfidence in their scientific knowledge. Among flat earth believers, a significant portion (24.7%) claimed to know the same or more than scientists, while this percentage was much lower (2.2%) among those who believed the earth isn’t flat.

The article discusses scientific literacy. Flat earthers don't score very high but think they know a lot about science. Who woulda thunk?

 

[hurtinbuckaroo]

What if the people who think they’ve debunked Dunning Kruger suffer from Dunning Kruger.

 

[annewimsey]

I prefer The Peter Principle, it explainsEverything IMO

 

[Shadowy Man]

Flat-earthers are overconfident about their own scientific knowledge but exhibit low scientific literacy, study finds

Furthermore, the researchers found that flat earth believers exhibited higher levels of overconfidence in their scientific knowledge. Among flat earth believers, a significant portion (24.7%) claimed to know the same or more than scientists, while this percentage was much lower (2.2%) among those who believed the earth isn’t flat.

The article discusses scientific literacy. Flat earthers don't score very high but think they know a lot about science. Who woulda thunk?

Well, you would have to in order to be a flat earther since the last thousand years of science doesn’t agree with you.