Martin's Paradox

by Michael T. Martin
ASBA Research Analyst

An Arizona Daily Star newspaper editorial (March 10, 2002) stated:

"None of the educators adequately explain a troubling aspect of the test scores: Performance tends to decline on the tests as the kids get older. Fifty-seven percent of the state's third-grade students passed the math test, but only 41 percent of the 5th graders, and a pathetic 18 percent of 8th graders."

The Star is not alone in making this allegation. The Economist magazine made a similar allegation in its cover story about national test scores two years ago (April 1-7, 2000). However, the allegation that public school students do less well in mathematics the longer they stay in public schools is a hoax. Student achievement from grade to grade in Arizona, and nationally, shows a high level of increasing achievement.

The allegation that students in Arizona (and nationally) are failing to learn middle school mathematics, allegedly because the curriculum "is a mile wide and an inch deep" is also a hoax. In fact, middle school students in Arizona and across the country are learning mathematics so well that that test scores could hardly be higher. The problem is not that the kids don't understand math, the problem is that the adults don't understand arithmetic.

There is a simple arithmetical explanation of why student test scores decline as student achievement increases, which I named Martin's Paradox: the better your educational system, the more students will score below average. That, in turn, results in test scores declining the more students learn because of the way test scores are "standardized."

Martin's Paradox occurs because as the educational system improves, and as student learning improves, the simple arithmetic of calculating student average scores results in more and more students scoring below average. Even though every test score improves, more able students will improve more than less able students. The higher scores of more able students (coupled with a reduction in very low test scores) results in a skewed distribution of scores.

This skew creates a well-known arithmetical consequence: the average score ("mean") will move above the middle score ("median") even as the median moves above the highest concentration of scores ("mode"). Which means more and more scores will fall below this higher average. More test scores farther and farther below average is exactly what you want an excellent school system to produce.

In summary form:

1. Before students are taught a course, knowledge of the subject will tend to be randomly distributed among the students: most will know a little about the subject, a few will know nothing and a few will know a lot, forming a bell curve distribution of knowledge.
2. This random distribution of knowledge implies that the mean, median and mode of any measure of knowledge will be in the same central location of a bell curve distribution.
3. At the end of the course, subject knowledge will tend to form a skewed distribution. Students who formerly knew nothing about the subject will have absorbed approximately the same level of knowledge as the majority of the class (therefore the lower tail of the distribution will be attenuated). But some of the more talented individuals, some of whom were in the middle of the original distribution, will absorb a significantly higher level of knowledge and thus score well above the rest of the class. This will enhance the upper tail of the distribution.
4. In the pre-course bell curve distribution, approximately half the students scored below average, but in the post-course skewed distribution the average score will be substantially above the median, which will be substantially above the mode (this is an empirical outcome dependent on the degree of learning).
5. The better you teach the students, the greater will be the increase in the mean above the median and the median above the mode, and therefore the more students will score below average (Martin's Paradox). This occurs because in the original distribution the lower tail of knowledge included very bright students who simply had not been taught the information, but who now score very high; and because students who knew some of the knowledge included less able students who will show only an incremental increase in knowledge from the course.
6. When these raw scores are standardized, the increase in the mean will disappear because the process of standardization, by definition, sets the mean to the standard.
7. Ipso facto, since the raw mean was substantially above the raw median which was substantially above the raw mode, the process of standardization will place the standardized scores of those students in the mode substantially below the median and place the median substantially below the mean.
8. Consequently, by definition, the standardized scores of the mode and median will be shifted farther below the standard the greater the raw mean was above the raw median and mode.
9. If the students were taught nothing in the course, then their post-course scores would be the same as their pre-course scores, and the mean, median and mode of a poor educational outcome would all be the same (in the center of the bell curve) both before and after standardization.
10. However, if the students learned a substantial amount of knowledge in the course, then their raw mean would be substantially above the raw median which would be substantially above the mode, and therefore the better your educational system the greater the standardized scores will decline because the more students score below average the more they will necessarily have scores below the standard.
11. Additionally, if knowledge tends to accumulate over time or over grade levels, then the same process will occur over time and over grade levels, and therefore standardized scores over the years will decline, and standardized scores over grade level increases will decline. The greater the decline, the more education has been successful.
12. This is a natural process that is independent of the subject taught, the test questions used, or the students compared.

Thus the degree of student learning is best expressed in terms of the skew in student scores. The greater the skew, the more the low scores have attenuated and the high scores have been enhanced, the better the educational system but the more scores will be below average. An actual example of this shown in the The Arizona Instrument to Measure Standards (AIMS) Mathematics Test

Another, more common, consequence of skewed test scores is explained in Martin's Comparadox, including an example from the The NAEP Reading test.