Quantifying not too easy and not too hard

[lpetrich]

How Wrong Should You Be? - Scientific American Blog Network

My best friend in college was a straight-A student, an English major. In part he got all A’s because he is whip smart—his essays were systematically better than everyone else’s. But the other reason was that he refused to enroll in a course unless he was certain he would ace it. Consequently, he never really challenged himself to try something beyond his comfort zone. I, on the other hand, was not a straight-A student. My first semester I took atomic physics with Professor Delroy Baugh, self-proclaimed “Laser Guy.” I’d never taken a physics course before in my life, and as a reward for my willingness to transcend my comfort zone I received a D.

Somewhere between the two of us lies a sweet spot: if you only ever get 100 percent on your tests, they aren’t hard enough. If you never get above 50 percent, you’re probably in the wrong major. The trick is to be right enough, but not so right that you never allow yourself the opportunity to be wrong.

So, how wrong should you be?

The Eighty Five Percent Rule for Optimal Learning | bioRxiv

Researchers and educators have long wrestled with the question of how best to teach their clients be they human, animal or machine. Here we focus on the role of a single variable, the difficulty of training, and examine its effect on the rate of learning. In many situations we find that there is a sweet spot in which training is neither too easy nor too hard, and where learning progresses most quickly. We derive conditions for this sweet spot for a broad class of learning algorithms in the context of binary classification tasks, in which ambiguous stimuli must be sorted into one of two classes. For all of these gradient-descent based learning algorithms we find that the optimal error rate for training is around 15.87% or, conversely, that the optimal training accuracy is about 85%. We demonstrate the efficacy of this 'Eighty Five Percent Rule' for artificial neural networks used in AI and biologically plausible neural networks thought to describe human and animal learning.

That number is from a theoretical argument that yielded (1/2) * (1 - erf(1/sqrt(2))) ~= 0.1587. It was tested with some numerical experiments in machine learning.

In academic terms, the appropriate level of difficulty would be B or B+, even if they don't look as good on one's record as A's.

 

[steve_bank]

I believe you are as only as good as the problems and challenges you face.

It takes failure to complete the process.

Over the last 15 or so years I have observed new college grads don't seem to have the same problem solving skills of previous generations.

If they are confronted with something beyond them if they can't find an app or something online they do not know what to do.

It is a generalization, but something others have observed. Too much reliance on computers in college.

I listened to a business owner in an interview. He new high school grads entering the work force do not have the self organizing capacity previous generations. The show pointed to kids having on over structured and organized life. No time and space for independent exploration.

 

[ronburgundy]

The general point about the need for some degree of challenge and failure is valid. However, the specific "85 percent rule" doesn't likely have much valid application to most actual human learning. First, that number is dependent upon the specific type of learning, which in was a binary classification task. That is a rather narrow form of learning that doesn't have much in common with something like learning complex causal models in science, where there are countless ways (rather than just one) that a learner can be wrong and gradations of being wrong (e.g., understanding 99% of the causal model to understanding 0% of it). Secondly, it completely ignores human motivations and emotions which vary greatly between persons and contexts and play a massive role in how negative feedback (being wrong) impacts learning, both by impacting how a learner regulates their future learning efforts and strategies and because emotional responses to failure and success have direct impacts on information processing and memory.

It is also important to bear in mind that getting an A in a college course does not mean that the student did not have productive failures.
In most courses, when can get an 85% on Exams and graded assignments and yet still get an A in the course, due either to grade curving or due to points given for attendance and effort where you get 100% just for showing up or completing some assignments. What matters is that there is still some difficulty and failures, not what the ultimate grade was.

Finally, being informed about exactly when and how you failed is critical for the benefits of such failures. Giving the wrong answer on a Exam will harm your future learning if you not informed that specific answer was wrong and how it was wrong. For example, if you wrongly choose option B on a multiple choice exam, but you are not told that B was wrong then giving that wrong answer actually makes that wrong answer stronger and more likely to be given in the future or to impact other related learning. Even if you told that B was wrong, if you don't know what was right, then you don't benefit as much from the experience. This relates to the simplistic binary categorization tasks used in the machine learning in the OP, where there are only two options, so any known failure inherently informs the learner what the correct answer should have been.

The bottom line is that we cannot draw valid inferences about the pre-post gains in knowledge and skills from a course simply by what grade the student got, and the "85"% rule" doesn't mean much beyond the narrow context of machines learning binary categories that it was developed with.