Standard Tests And Bias

[Metaphor]

But it was supposed to test ability to do long division, not to pick the right answer (using multiplication and elimination). Point proven.

"Look's like somebody's gotta case of the 'supposed to's'" (A The Simpsons reference from when it was superlative 25 years ago)

First, multiplication and division are related concepts. As I remember from year 8, you can multiply by the reciprocal to divide fractions.

Second, your knowledge of what the right answer looks like in a long division is long division knowledge.

Third, no single question measures your long division skill.

Fourth, questions can be measured for how well they correlate with other questions that are also designed to measure to the trait in question.

Jarhyn explained it more elaborately (better):

Jarhyn did nothing of the kind. In fact, his example showed how one needs multiple pieces of mathematical knowledge to identify the correct answer. It beggars belief that you think the two examples show what you think they show.

I even recall geometry test that asked for the distance from x to y (no multiple choice) involving a curve and some planar coordinates... at sea with applying the formula for the parabola, I drew it roughly and measured the distance, which was really close to a rational number of units so I used that number. It was correct, and took a fraction of the time that calculating it would have taken, even if I knew what I was doing.

In high school, getting the correct number was almost never enough for full marks. You had to show your work. But even where you don't in the example above, you got the correct answer to the question because you understood the subject matter enough to know what was being asked.

Test-taking is a skill (or call it a talent) that can have a lot - or almost nothing - to do with the actual subject matter that the test designer had in mind.

Test-taking is a skill, sure. That does not mean standardised testing measures only test-taking skill. That's an idiotic claim unsupported by evidence.

 

[Lumpenproletariat]

Well, you can't ignore genetic confounding. Here's an example. In a legal case where a delivering doctor seriously harmed a child during delivery - such that the child has severe motor and cognitive deficits - that child can make a claim for future income loss. But how do you determine probable future income loss for someone with no work history; who has never worked and never will? You look at the parents. The probably is that a child will do as well or slightly better than the parents. If the parents are professionals, it is likely the child would have been a professional. If the parents are low-income, the same is likely for the child. Once a geneticist was rather blunt to me: if the parents are dim than it's not surprising for the child to be dim.

Your example represents a very small minority. Most people are not that dim, they are able to do productive work.

What about the working poor who work full time, long hours doing productive work for pittance rates of pay? Not because they are not useful to society, the business, not because their work is not essential, but simply because as individuals in the position they are in, doing work that has no prestige or status attached, they lack the power to negotiate better pay and conditions?

I.e., they are uncompetitive in the market and thus have less supply-demand value.

The solution for them is to learn more and become more valuable, including for society to make more learning opportunities available to them. And for society to better reward those who have more knowledge and abilities, hire more of them at competitive wage level, and in other ways get them involved in putting their abilities to a socially-beneficial use.

The solution is not to artificially prop up their wage level in the low-value uncompetitive jobs they are already in, thus scapegoating their employer as the villain and overpaying uncompetitive workers for their low-value labor, incentivizing them to remain in those uncompetitive jobs and incentivizing the employer to reduce production to save on cost.

 

[Elixir]

Test-taking is a skill, sure. That does not mean standardised testing measures only test-taking skill.

Did I say it did? Read this again and tell me. I even bolded part of it so you might pick up on it:
"Test-taking is a skill (or call it a talent) that can have a lot - or almost nothing - to do with the actual subject matter that the test designer had in mind."

That's an idiotic claim unsupported by evidence.

...which is one reason I didn't make that claim (illustrating why I don't engage with you, Meta - you are either dishonest or unable to read for comprehension).
Meanwhile, your claim that knowing that getting a product ending in 1 by multiplying by 7 requires a 3 is the same thing as doing long division, is ludicrous.
Knowing that 2x4 yields a product greater than 7 doesn't do it either. And those were the only requisites for divining the multiple choice answer to (as opposed to solving) the long division problem in my example.

 

[Metaphor]

Did I say it did? Read this again and tell me.

Many people who are against standardised tests say it often. That it measures only 'test taking skill', or 'privilege'.

I even bolded part of it so you might pick up on it:

"Test-taking is a skill (or call it a talent) that can have a lot - or almost nothing - to do with the actual subject matter that the test designer had in mind."

And yet the mere existence of test-taking ability as a concept is enough for many people who are against standardised testing to dismiss all standardised tests (including people on this board who humblebrag about how they were 'good at taking tests')

Meanwhile, your claim that knowing that getting a product ending in 1 by multiplying by 7 requires a 3 is the same thing as doing long division, is ludicrous.
Knowing that 2x4 yields a product greater than 7 doesn't do it either. And those were the only requisites for divining the multiple choice answer to (as opposed to solving) the long division problem in my example.

I said

your knowledge of what the right answer looks like in a long division is long division knowledge.

But if you think long division is a required thing to test (I was never taught long division and I survived the world), and you think the knowledge you used to get the right answer does not demonstrate that knowledge (in your example), then design an item that does test long division, like every maths test I ever took in high school that said 'show your work'.

 

[DBT]

Well, you can't ignore genetic confounding. Here's an example. In a legal case where a delivering doctor seriously harmed a child during delivery - such that the child has severe motor and cognitive deficits - that child can make a claim for future income loss. But how do you determine probable future income loss for someone with no work history; who has never worked and never will? You look at the parents. The probably is that a child will do as well or slightly better than the parents. If the parents are professionals, it is likely the child would have been a professional. If the parents are low-income, the same is likely for the child. Once a geneticist was rather blunt to me: if the parents are dim than it's not surprising for the child to be dim.

Your example represents a very small minority. Most people are not that dim, they are able to do productive work.

What about the working poor who work full time, long hours doing productive work for pittance rates of pay? Not because they are not useful to society, the business, not because their work is not essential, but simply because as individuals in the position they are in, doing work that has no prestige or status attached, they lack the power to negotiate better pay and conditions?

I.e., they are uncompetitive in the market and thus have less supply-demand value.

The solution for them is to learn more and become more valuable, including for society to make more learning opportunities available to them. And for society to better reward those who have more knowledge and abilities, hire more of them at competitive wage level, and in other ways get them involved in putting their abilities to a socially-beneficial use.

The solution is not to artificially prop up their wage level in the low-value uncompetitive jobs they are already in, thus scapegoating their employer as the villain and overpaying uncompetitive workers for their low-value labor, incentivizing them to remain in those uncompetitive jobs and incentivizing the employer to reduce production to save on cost.

The problem is not that workers on the lower end of the scale are uncompetitive, just that individually they lack leverage. Being easy to hire and fire, they have no ability to negotiate a better deal. Which is where collective bargaining comes in.......

 

[Lumpenproletariat]

I.e., they are uncompetitive in the market and thus have less supply-demand value.

The solution for them is to learn more and become more valuable, including for society to make more learning opportunities available to them. And for society to better reward those who have more knowledge and abilities, hire more of them at competitive wage level, and in other ways get them involved in putting their abilities to a socially-beneficial use.

The solution is not to artificially prop up their wage level in the low-value uncompetitive jobs they are already in, thus scapegoating their employer as the villain and overpaying uncompetitive workers for their low-value labor, incentivizing them to remain in those uncompetitive jobs and incentivizing the employer to reduce production to save on cost.

The problem is not that workers on the lower end of the scale are uncompetitive, just that individually they lack leverage.

Same thing. Uncompetitive, lacking leverage, easily replaceable by robots or cheap labor -- all different terms for the same thing.

Being easy to hire and fire, they . . .

Yes, because they're so easy to replace with robots or cheap labor. = uncompetitive or less valuable.

. . . they have no ability to negotiate a better deal.

Because of their lower value, less needed by the employer. That's what "lower value" and "less competitive" mean, and why they're paid less.

Which is where collective bargaining comes in.......

Right. Only uncompetitive crybabies need collective bargaining, to artificially prop up their bargaining power, rather than competing and being paid according to their value.

to return to the current topic -- The real solution for them is not to throw a tantrum, but to improve their capability, learn more, become more educated, and be able to pass tests of their ability. To be less replaceable also means to be more knowledgeable, which means able to pass tests and demonstrate their ability or knowledge.

 

[Loren Pechtel]

Standardized tests are a proxy for IQ. So it’d not be surprising that those with high SAT/ACT scores also had high GPA. But GPA is a less reliable indicator as it’s dependent on some subjective teacher grading and relation to high school peers.

What the study found is that amongst those with adequate ACT scores GPA was more predictive of success. Note that that means the students with a high GPA from a shitty school were basically excluded and thus no conclusions can be drawn about how useful it is without first filtering with with SAT/ACT.

Since you not read the report, you don't know what the study basically found. The study's goal was to look at which was a better predictor of college success - HS GPA or SAT/ACT. Naturally, it only included students who went to college. Any student who do not go to college should not have included. Neither you nor the authors had the information on why a student did not go to college. Your focus on high GPAs from shitty schools who did not go to college makes the assumption they applied but were not admitted to college because of their SAT/ACT score. Now, I wonder why you keep repeating in illogical point over and over when you do not mention high GPAs from good schools who did not go to college or any other HS GPA range from any type of school who did not go to college.

1) Keeping repeating that I didn't read it doesn't make it so. You have never explained how I quoted from something I didn't read.

2) You flunked statistics 101 here. Current school admissions take SAT or ACT strongly into consideration. Somebody who doesn't do well on them isn't getting into college. The study was of the graduation rate from college--and it should be obvious that to succeed or fail at college first requires getting into college. Thus the people with high GPAs but low SAT/ACT scores are not part of the sample. Thus the study can say nothing about them.

 

[Loren Pechtel]

But "understanding" the subject matter is the same as doing well on tests, or being able to, or able to answer questions about it, or to perform what the subject matter is about. The phrase "understanding the material" is meaningless if it doesn't mean being able to perform well on a test of one's knowledge of the subject matter.

Disagree. High school, a truly miserable English teacher. She had strong opinions about the books we were assigned to read and you would be marked down severely if you didn't agree--even though she often hadn't even expressed her opinion.

Anyway, she gave a lot of true/false questions. After a bit I learned to pay no attention to what I knew when answering them. I got about 95% by saying anything stated in a positive sense was true and anything stated in a negative sense was false. That was better than I could do in figuring out what she was thinking.

And not quite as relevant, but I bombed a calculus final because I made *one* mistake of memory. I got one sign wrong, then propagated the error into other formulas. Relevance to actually using it? The next class had a book on the requirements list--a book of formulas that we were allowed to use on tests. Including the very one I had goofed. In practice you almost always have reference sources available, what is important is understanding how to use those sources, not remembering them in the first place.

I'm a computer programmer. The editor provides helpful lookup information about what you're typing--because Microsoft perfectly well knows you're not going to memorize all the details of the libraries. Google is also of great help because the built-in stuff doesn't have nearly as good searching capability. Does that mean I don't know how to program?? No--someone who doesn't know how to program wouldn't understand what they are reading and how to apply it.

Or consider nearly 40 years ago. I'm in the office of a tropical disease specialist. After giving him the information he steps out of the room for about 20 minutes before coming back and writing a prescription. Does that mean he didn't know his field? No, it means he had to go find the right book to look up the drug resistances for the time & place I was bit to know what drugs would work. (And I had a similar experience close to 20 years ago, except he had the internet and found the information a lot faster. Somehow the first guy drove the malaria into hiding for 20 years but didn't kill it off--in that time the closest I had been to the malaria zone was Arizona.)

 

[Loren Pechtel]

So, I'm good at testing. Like, stupid good at it. Give me a multiple choice test with as many as six answers, and I will assure you I will get less than 1/2 of the questions wrong even if I don't actually know the answers.

If the test is for a single chapter on a subject, this number will be down as low as 1/3.

This is because most tests ask questions in a particular way: "which answer is the result of X+Y?", Wherein some basic knowledge will throw out two answers right away, one is going to be close "but probably wrong", and the remaining two allow for a test of one to true the other.

Most teachers, yes, because they're lazy about test questions. Unfortunately, none of my parent's old tests made it to this century because I can be pretty sure you would not perform that well against theirs. They were both very good at writing test questions that required you to combine two pieces of knowledge. Open book/open note, true/false or multiple choice, if you knew the material the questions were dead easy. If you didn't--I saw (I did almost all of my mother's grading) people do worse than random guessing. If you forgot a name or the like you could look it up. Don't understand the material and you were up shit creek.

 

[Loren Pechtel]

They should exist at all levels -- ideally. If it's not yet totally practical, then extend the opportunities as much as possible -- to more levels.

It should be possible for anyone to enroll into a rocket science program. Maybe not today, but eventually. The same program could be open to a 5-year-old (prodigy?), and also to an 80-year-old geezer. They would be tested and routed to wherever their appropriate learning level is, regardless of all the other differences. And this should be open to EVERYONE to apply and be accepted, and then be routed to where they can best learn at their level.

The narrower the range of abilities of the students in a class the better off the students are.

OK, but there should be a rocket-science "class" to everyone, based on their aptitude. So, no one should be excluded from the program. Certain ones who are poor performers at first might improve, based on the testing. So any of them could advance if they show improvement, based on the testing.

Also, there's a place here for some of the more advanced students to participate in teaching the less advanced ones, so there needn't be total segregation of the more able from the less able.

There comes a point where it's not a productive use of resources.

On the other hand, I would like to see at least 90% of classes abolished. The world no longer needs them.

For most subjects we should break the classes down into substantially smaller units, then take each such unit and a group of good teachers of that material and make up a "class". Several explanations of the material, both video and text, written at various paces, and a bunch of homework questions. The teachers get repurposed as tutors, helping the students who are stuck and reporting in true misunderstandings (as opposed to simply students being lazy) so material can be added addressing those misunderstandings. All of the material is posted on a government website, freely downloadable.

Obviously, you're still going to need some early classes (they're going to have to learn to read before they can use the computerized stuff) and you still need actual classes for things which are hands-on or interactive and you'll need actual classes for things which a computer can't reasonably determine right or wrong. (A computer can do a good job of determining that you used English properly in your essay, but can't tell if you actually wrote bullshit.)

 

[Loren Pechtel]

"Look's like somebody's gotta case of the 'supposed to's'" (A The Simpsons reference from when it was superlative 25 years ago)

First, multiplication and division are related concepts. As I remember from year 8, you can multiply by the reciprocal to divide fractions.

Second, your knowledge of what the right answer looks like in a long division is long division knowledge.

The problem here is that the real world is fill in the blank, not multiple choice. The math shortcuts discussed here help you pick the correct answer from the list but if you don't have the list and simply need the answer none of them are of any value.

 

[Metaphor]

"Look's like somebody's gotta case of the 'supposed to's'" (A The Simpsons reference from when it was superlative 25 years ago)

First, multiplication and division are related concepts. As I remember from year 8, you can multiply by the reciprocal to divide fractions.

Second, your knowledge of what the right answer looks like in a long division is long division knowledge.

The problem here is that the real world is fill in the blank, not multiple choice. The math shortcuts discussed here help you pick the correct answer from the list but if you don't have the list and simply need the answer none of them are of any value.

Then formulate items that are not multiple choice. But don't cry that tests measure test taking ability and must always be hopelessly biased, and therefore eliminated.

 

[fromderinside]

What seems to be the topic here is not bias but ineffectual test design lacking clear operational definition guidance.

 

[laughing dog]

Since you not read the report, you don't know what the study basically found. The study's goal was to look at which was a better predictor of college success - HS GPA or SAT/ACT. Naturally, it only included students who went to college. Any student who do not go to college should not have included. Neither you nor the authors had the information on why a student did not go to college. Your focus on high GPAs from shitty schools who did not go to college makes the assumption they applied but were not admitted to college because of their SAT/ACT score. Now, I wonder why you keep repeating in illogical point over and over when you do not mention high GPAs from good schools who did not go to college or any other HS GPA range from any type of school who did not go to college.

1) Keeping repeating that I didn't read it doesn't make it so. You have never explained how I quoted from something I didn't read.

I did. Apparently you did not read that either. You quoted the summary not the article. IF you had read the article with even the basic level of reading comprehension, you would not have persisted in your misinterpretation about the effects of the high schools.

2) You flunked statistics 101 here. Current school admissions take SAT or ACT strongly into consideration. Somebody who doesn't do well on them isn't getting into college. The study was of the graduation rate from college--and it should be obvious that to succeed or fail at college first requires getting into college. Thus the people with high GPAs but low SAT/ACT scores are not part of the sample. Thus the study can say nothing about them.

At least you dropped the qualifier of "shitty school" in your response. But you still flunked logic 101 and statistics 101. First, you don't know that there are people with High GPAs but low SAT/ACT scores. Second, you don't know why anyone who might have been the high GPA but low SAT/ACT did not go to college - they may not have applied or they have been admitted but not attended for any myriad of reasons. Third, there is no way to include in a study of college success, people who do not attend college. So, according to your logic, we cannot accept the results from any study about predictors of college success.

 

[DBT]

Same thing. Uncompetitive, lacking leverage, easily replaceable by robots or cheap labor -- all different terms for the same thing.

Being easy to hire and fire, they . . .

Yes, because they're so easy to replace with robots or cheap labor. = uncompetitive or less valuable.

. . . they have no ability to negotiate a better deal.

Because of their lower value, less needed by the employer. That's what "lower value" and "less competitive" mean, and why they're paid less.

Which is where collective bargaining comes in.......

Right. Only uncompetitive crybabies need collective bargaining, to artificially prop up their bargaining power, rather than competing and being paid according to their value.

to return to the current topic -- The real solution for them is not to throw a tantrum, but to improve their capability, learn more, become more educated, and be able to pass tests of their ability. To be less replaceable also means to be more knowledgeable, which means able to pass tests and demonstrate their ability or knowledge.

You keep repeating the same fallacies regardless of how many times they are shot down by studies, stats and examples. Low pay workers are not paid a pittance because they have 'low value' but because their position, without collective bargaining, does not give them the leverage to negotiate better rates of pay and conditions. The work they do is essential. Somebody has to do the cleaning, somebody has to flip burgers and serve customers....not everyone can be doctors or lawyers....

 

[Jarhyn]

I did. Apparently you did not read that either. You quoted the summary not the article. IF you had read the article with even the basic level of reading comprehension, you would not have persisted in your misinterpretation about the effects of the high schools.

2) You flunked statistics 101 here. Current school admissions take SAT or ACT strongly into consideration. Somebody who doesn't do well on them isn't getting into college. The study was of the graduation rate from college--and it should be obvious that to succeed or fail at college first requires getting into college. Thus the people with high GPAs but low SAT/ACT scores are not part of the sample. Thus the study can say nothing about them.

At least you dropped the qualifier of "shitty school" in your response. But you still flunked logic 101 and statistics 101. First, you don't know that there are people with High GPAs but low SAT/ACT scores. Second, you don't know why anyone who might have been the high GPA but low SAT/ACT did not go to college - they may not have applied or they have been admitted but not attended for any myriad of reasons. Third, there is no way to include in a study of college success, people who do not attend college. So, according to your logic, we cannot accept the results from any study about predictors of college success.

According to your analysis of his logic. He might not have thought it through to that point. Though there is a way to determine that. You would need a random longitudinal study of high school students across the country followed through senior year, starting Senior year GPA, post-slide GPA, SAT/ACT scores, and tracking college admissions, applications, and arrival.

It would require interior department of education data, and be quite expensive, but it's a doable study.

 

[laughing dog]

I did. Apparently you did not read that either. You quoted the summary not the article. IF you had read the article with even the basic level of reading comprehension, you would not have persisted in your misinterpretation about the effects of the high schools.

2) You flunked statistics 101 here. Current school admissions take SAT or ACT strongly into consideration. Somebody who doesn't do well on them isn't getting into college. The study was of the graduation rate from college--and it should be obvious that to succeed or fail at college first requires getting into college. Thus the people with high GPAs but low SAT/ACT scores are not part of the sample. Thus the study can say nothing about them.

At least you dropped the qualifier of "shitty school" in your response. But you still flunked logic 101 and statistics 101. First, you don't know that there are people with High GPAs but low SAT/ACT scores. Second, you don't know why anyone who might have been the high GPA but low SAT/ACT did not go to college - they may not have applied or they have been admitted but not attended for any myriad of reasons. Third, there is no way to include in a study of college success, people who do not attend college. So, according to your logic, we cannot accept the results from any study about predictors of college success.

According to your analysis of his logic. He might not have thought it through to that point. Though there is a way to determine that. You would need a random longitudinal study of high school students across the country followed through senior year, starting Senior year GPA, post-slide GPA, SAT/ACT scores, and tracking college admissions, applications, and arrival.

It would require interior department of education data, and be quite expensive, but it's a doable study.

That would help, but I guarantee if it came up with the same result (HS GPA better predictor than SAT/ACT), some flaw (imaginary or trivial) would still be adduced.

And, no study can determine how someone who did not go to college would have done in college. So, at its core,LP's objection is illogical. And that abstracts from the questions of whether someone who had applications rejected would have been accepted and graduated at some other school or the student with a good SAT/ACT who did not go to college would have graduated if they had gone or any other myriad of possibilities.

And all of this abstracts from my original point that there is research to support the dropping of SAT/ACT scores as a requirement. Whether or not someone agrees with that decision is not relevant to the actuality that schools have sound reasons for doing so.

I also find it fascinating that so many people who clearly have no clue about how many institutions analyze applications for admission feel they have sufficient expertise to opine on those policies or how those policies dovertail with their goals.

 

[Trausti]

And all of this abstracts from my original point that there is research to support the dropping of SAT/ACT scores as a requirement. Whether or not someone agrees with that decision is not relevant to the actuality that schools have sound reasons for doing so.

$$$

 

[Loren Pechtel]

I did. Apparently you did not read that either. You quoted the summary not the article. IF you had read the article with even the basic level of reading comprehension, you would not have persisted in your misinterpretation about the effects of the high schools.

2) You flunked statistics 101 here. Current school admissions take SAT or ACT strongly into consideration. Somebody who doesn't do well on them isn't getting into college. The study was of the graduation rate from college--and it should be obvious that to succeed or fail at college first requires getting into college. Thus the people with high GPAs but low SAT/ACT scores are not part of the sample. Thus the study can say nothing about them.

At least you dropped the qualifier of "shitty school" in your response. But you still flunked logic 101 and statistics 101. First, you don't know that there are people with High GPAs but low SAT/ACT scores. Second, you don't know why anyone who might have been the high GPA but low SAT/ACT did not go to college - they may not have applied or they have been admitted but not attended for any myriad of reasons. Third, there is no way to include in a study of college success, people who do not attend college. So, according to your logic, we cannot accept the results from any study about predictors of college success.

According to your analysis of his logic. He might not have thought it through to that point. Though there is a way to determine that. You would need a random longitudinal study of high school students across the country followed through senior year, starting Senior year GPA, post-slide GPA, SAT/ACT scores, and tracking college admissions, applications, and arrival.

It would require interior department of education data, and be quite expensive, but it's a doable study.

Yes, this is how the study would have to be done and it would be a lot harder to do than the referenced study.

 

[Loren Pechtel]

That would help, but I guarantee if it came up with the same result (HS GPA better predictor than SAT/ACT), some flaw (imaginary or trivial) would still be adduced.

I don't discard all research I disagree with. It's just the social sciences are full of "research" with gaping holes in it.

And, no study can determine how someone who did not go to college would have done in college. So, at its core,LP's objection is illogical. And that abstracts from the questions of whether someone who had applications rejected would have been accepted and graduated at some other school or the student with a good SAT/ACT who did not go to college would have graduated if they had gone or any other myriad of possibilities.

No--while the question obviously can't be answered completely short of trying it you would at least see the size of the problem being dealt with. As it is we have a glaring unknown that is being tiptoed past.

I also find it fascinating that so many people who clearly have no clue about how many institutions analyze applications for admission feel they have sufficient expertise to opine on those policies or how those policies dovertail with their goals.

The problem is they have repeatedly demonstrated they desire to discriminate, thus we are highly suspicious of anything which appears to favor such discrimination.