|A Joint Position
Statement by the Canadian Psychological Association and the Canadian Association of School
Psychologists on the Canadian Press Coverage of the Province-wide Achievement Test Results
Marvin L. Simner, Ph.D.
|Executive Summary | Background | Position Statement | Recommendations
Executive SummaryEach spring teachers throughout Canada are required to administer a series of provincially-mandated tests to the students in their classes. Approximately six months later the results are made available to the public. It is now common practice for the press to report the results, to rank the schools according to the results, and to invite the public to engage in a school-by-school comparison of the rankings. It is also common practice for the press, in commenting on the poor performances displayed by certain schools, to place the blame for these performances largely, if not solely, on the schools themselves.
Our concern over this practice is with the failure on the part of the press to acknowledge the many other factors aside from schooling that are known to influence test performance. These include, but are not limited to, family stability, parental involvement and expectations for student success in school, early and ongoing home stimulation, and student motivation, student absenteeism, as well as student capacity for learning. Because students are not randomly assigned to schools and because schools have little or no control over the majority of these factors, any attempt to place the blame for poor test performance on the schools alone without giving proper consideration to each of these other factors is problematic at best and misleading at worst. Hence, by taking such a narrow position when dealing with an issue as complex as the cause of students test performances, we believe the public is being misinformed on a matter that is extremely important to the operation of schools and therefore to the education and well being of children.
Our greatest concern, however, is over the possibility that this singular emphasis on the schools as the cause of students test performances could generate considerable harm by placing unwarranted pressure on teachers, administrators, and ultimately on the students themselves to increase test scores or risk losing status within the community. Indeed, there is growing evidence that such harm is already occurring in the United States from similar comments in the US press concerning the poor performance of certain schools on the state-mandated tests.
With the hope of preventing similar situations from occurring in Canada, and in line with the view expressed by the Ontario Education Quality and Accountability Office, which is responsible for the development and scoring of the Ontario exams, it is our position that it is improper for the press to invite the public to compare schools based solely on the outcome of the mandated test results. We also recommend that in any future articles that deal with the results, in order to avoid misleading the public, the press should ensure that the public is fully informed of the various factors, in addition to schooling, that are likely to account for differences that may exist among schools.
Province-wide achievement testing of elementary and high school students is now the norm throughout Canada (Canadian Federation of Teachers, 1999). Although the nature of the tests and the grade levels tested varies from province to province, each province, including the territories, either administers or plans to administer some form of standardized test to all of its students on an annual or biennial basis. Whereas we share many of the concerns being voiced today over the appropriateness of such mandated system-wide testing (see, for example, Froese-Germain, 1999; Haladyna, Nolen, & Hass, 1991; Herman & Golan, 1993; Herman, Abedi, & Golan, 1994; Jones & Whitford, 1997; Lomax, West, Harmon, Viator, & Madaus, 1995; Madaus, 1985; Paris, Lawton, Turner, & Roth, 1991; Rotberg, 1995; Salganik, 1985), our immediate concern is over the harm that might result from the misleading statements that are now appearing in the press regarding the findings.
In recent years it has become common practice for the press to report the test scores for individual schools, to rank the schools according to the scores, and to invite the public to engage in a school-by-school comparison of the rankings. The purpose of this invitation was perhaps best expressed in the following passage from a feature article in The Calgary Herald by Peter Cowley, coauthor of the 1999 Report Card on Albertas High Schools compiled by the Fraser Institute.
A similar message appeared in the Vancouver newspaper The Province on March 24, 1999, as part of a three-day feature story on the rankings of the British Columbia high schools (also prepared by the Fraser Institute). An even more forceful rendition of this message appeared in a lead editorial in The London Free Press on November 27, 1999.
Thus, according to these statements, it would seem that it is the schools that are largely responsible for the students test performances. Moreover, unless the schools with the poorest performing students are forced, through competition, to provide their students with a quality education, the students will continue to receive only a mediocre learning experience. Our concern over this material is with the failure on the part of the authors to acknowledge the many other factors aside from schooling that are known to influence test performance. These include, but are not limited to, family stability, parental involvement and expectations for student success in school, early and ongoing home stimulation, and student motivation, student absenteeism, as well as student capacity for learning (Christenson, Rounds, & Gorney, 1992; Grolnick & Ryan, 1989; Jeynes, 1999; Keith, Keith, Quirk, Sperduto, Santillo, & Killings, 1998; Lytton & Pyryt, 1998; McLean, 1997; Scarborough & Dobrick, 1994). Because students are not randomly assigned to schools and because schools have little or no control over the majority of these factors, any attempt to place the blame for poor test performance on the schools alone without giving proper consideration to each of these other factors is problematic at best and misleading at worst. Hence, by taking such a narrow position when dealing with an issue as complex as the cause of students test performances, we believe the public is being misinformed on a matter that is extremely important to the operation of schools and therefore to the education and well being of children.
The same can be said of statements appearing in articles that comment on the year-to-year changes in test scores obtained by individual schools. The Globe and Mail, for example, highlighted a single school in the Toronto Catholic School District that reported a 48% gain in reading performance and attributed this gain solely to "a concerted effort by the entire school to improve" (Galt, 1999). Similarly, the Readers Digest, in an article entitled "When teachers sell kids short" (Nikiforuk, 1997), attributed the gains made by one school in the North York district of Ontario to an "action plan [that] included more direct and sequenced teaching of phonics, spelling and grammar." Needless to say, by singling out specific schools that showed considerable improvement from one year to the next and by attributing this improvement only to efforts undertaken by the schools themselves, other schools that failed to show the same level of improvement, by implication, are assumed to be at fault for not engaging in similar efforts. The problem here too is that any gains or losses over time displayed by individual schools are difficult if not impossible to interpret without access to further information. The following example from The London Free Press illustrates the nature of this problem.
In the lead editorial mentioned above the Free Press claimed that 12 schools in the London and District Catholic School System "leapfrogged ahead" between 1998 and 1999 as a result of a "SWAT team of roving teacher specialists." If the editorial writer had considered the actual data, as well as some demographic findings available from the Catholic Board on these 12 schools, a somewhat different picture would have emerged. Take for instance reading. Aside from the fact that, according to the data, only seven of the 12 schools made gains large enough that they could be attributed to something other than chance, close inspection of the demographic findings revealed that these seven successful schools may have differed in important ways from the five other schools. In particular, among the successful schools there was a decrease between 1998 and 1999 in the average class size per school, whereas the opposite was true among the other schools. Thus it could be that, due to a smaller teacher-to-pupil ratio, the teachers in the successful schools had more time to spend with their students. Furthermore, in 1999 there were fewer males in the seven successful schools and more males in the five other schools. Also, between 1998 and 1999 there was an average drop of 12% in the number of children for whom English is a second language (ESL) among five of the successful schools, whereas in the five other schools, this drop was less than 2%. We mention these last two points because males as well as nonnative English speakers typically perform more poorly on reading exams than do females and native English speakers. In essence, this combination of smaller class sizes, fewer males, and fewer ESL children among the seven successful schools may have been sufficient to account for the differences between the two groups of schools independently of the presence of the "SWAT team."
As a further illustration of the need by the Free Press to have gathered more information prior to reaching any conclusions, consider what happened to the province-wide scoring of the exams between 1998 and 1999. According to a memo dated November 15, 1999, from the Education Quality and Accountability Office (EQAO), which is responsible for the development and scoring of the Ontario exams, the scoring procedures used in 1998 differed from the scoring procedures used in 1999. In 1998 the exam markers employed a four point scale (Level 1, 2, 3, 4) to determine each students achievement level, whereas in 1999 they used a seven point scale (Level 1, 1+, 2, 2+, 3, 3+, 4). The reason for this change was that the seven point scale permitted greater latitude in scoring when there was some uncertainty over whether a student deserved, for instance, a score at level 2 or 3. However, when EQAO calculated the final mark for each student in 1999, "the seven-point scale was collapsed into a four-point reporting scale (and) the levels assigned to students work were determined by combining the marks in each of the "plus" categories with those in the next higher level." In other words, whenever a "plus" appeared on a students record in 1999, the student automatically received a final score that was rounded up to the next higher level. Because rounding up did not take place in 1998, and because some schools may have profited more than others from this procedure, it is difficult to know whether the gains displayed by certain schools in the London and District Catholic School System between 1998 and 1999 should have been attributed to efforts undertaken by the schools alone (as stated in the editorial) or, instead, should have been attributed to an artifact that resulted from this change in the scoring procedure.
In summary, as is the case when attempting to determine the cause of differences that occur between schools within a given year, any attempt to determine the cause of differences that occur within schools from year-to-year is equally problematic without giving proper consideration to further information. The RAND Institute of Education and Training had the following to say about this matter.
Beyond this issue of test interpretation however, our greatest concern is over the harm that might result as a function of these misleading statements. As professional associations dedicated to the improvement of the mental health of children as well as adults, we are especially troubled by the possibility that this singular emphasis on the schools as the cause of the students test performances could place unwarranted pressure on teachers, administrators, and ultimately, the students themselves to increase test scores or risk losing status within the community. This possibility is clearly illustrated in the situation that is now unfolding in the United States. Consider Michigan, for example, where rankings similar to those that are appearing in Canada have been reported in the press for many years.
Consider also the following press reports that deal with recent situations in New York, Texas, Massachusetts, Maryland, Ohio, and Connecticut. In each instance school personnel were indicted, suspended, or are under investigation for improper behavior, and in each instance the responsible factor was the increased pressure on the schools that resulted from the rankings.
On a more personal note, the following quotation from a recent article in The New York Times by a fifth grade teacher in California no doubt captures the dilemma that many educators in the United States are now facing as the result of these rankings.
Additional commentary and evidence bearing on the harm that these rankings are causing in the United States can be found in articles by Lomax, West, Harmon, Viator, and Madaus (1995), Paris, Lawton, Turner, and Roth (1991), and Smith (1991). In summarizing much of this evidence, Rotberg (1995) had the following to say about holding educators accountable through test scores in order to bring about change in the educational system.
In light of the situation that is now taking place in the United States, and with the hope of avoiding a similar situation in Canada, we are in full agreement with the following statements by the Ontario Education Quality and Accountability Office. The first statement is from a handbook intended for parents; the second statement is from a handbook intended for educators.
In short, according to EQAO, the major purpose of the province-wide exams is to provide individual schools, and boards, with an opportunity to gauge their own year-to-year progress in meeting the standards established by the Ministry of Education. The exams are not intended to be used as yardsticks for measuring the performance of one school in relation to another. Hence, in supporting this position by EQAO, we consider it improper for the press to invite the public to contrast the performances of one school against another solely on the basis of the exam scores.
It is also our view that if the press continues to report, as well as interpret the findings from the exams, this action by itself automatically makes the press a user of educational test results. Therefore we feel that the press should abide by the same sets of ethical provisions that pertain to other users of this material. These provisions appear in such documents as the Principles for Fair Student Assessment Practices for Education in Canada (1993), Standards for Educational and Psychological Testing (AERA, APA, NCME, 1985, 1999), and Guidelines for Educational and Psychological Testing (CPA, 1987). Among the provisions are at least two that apply directly to the present situation.
In keeping with these provisions we recommend that in any future articles that report or interpret the findings, the press should ensure that the public is fully informed of the various factors, in addition to schooling, that are likely to account for differences that may exist among schools. As mentioned above these include, but are not necessarily limited to, family stability, parental involvement and expectations for student success in school, early and ongoing home stimulation, student motivation, student absenteeism, and student capacity for learning.
As a further means of avoiding any misunderstanding on the part of the public we also recommend that a note similar to the following should appear in a prominent position together with any school-by-school breakdown of the results.
In making these recommendations we also wish to make clear that it is not our intention to interfere with the freedom of the press. Nor is it our intention to discourage public debate over how best to educate children. We do feel, however, that the public is not well served when it is exposed to misleading information and unsupported speculation.
Appreciation is extended to the following members of the Working Group who offered many helpful comments on an earlier draft of this document: Sampo Paunonen, Nicholas Skinner, P. A. Vernon. Appreciation is also extended to Brian Reiser, Gary Rollman, and Phyllis Simner who, in addition to providing helpful comments, also supplied a number of key references.
AERA, APA, NCME. (1985). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
AERA, APA, NCME. (1999). Standards for educational and psychological testing. Washington, DC: American Psychological Association.
Bushweller, K. (1997). Teaching to the test. The American School Board Journal, September, 20.
Canadian Federation of Teachers. (1999). Province-wide Assessment Programs. Available on line at http://www.ctf-fce.ca/e/what/other/assessment/testing-main.htm.
Christenson, S. L., Rounds, T., & Gorney, D. (1992). Family factors and student achievement: An avenue to increase students success. School Psychology Quarterly, 7, 178-206.
Cowley, P. (1999, June 20). Comparing schools made possible. Calgary Herald, p. A10.
CPA. (1987). Guidelines for educational and psychological testing. Ottawa, ON: Canadian Psychological Association.
Daniszewski, H. (1999, May 4). Upping the score. London Free Press, p. A10.
EQAO. (1998a). Parent handbook 1997-1998. Toronto, Ontario: Queens Printer of Ontario.
EQAO. (1998b). Educators handbook. Toronto, Ontario: Queens Printer of Ontario.
Froese-Germain, B. (1999). Standardized testing: Undermining equity in education. Ottawa, ON: Canadian Teachers Federation.
Galt, V. (1999, November 26). Scarborough schools grade 3 posts sharp rise in test scores. The Globe and Mail.
Goodnough, A. (1999, December 9). New York city teachers nabbed in school-test cheating scandal. National Post, p. B1.
Grolnick, W. S., & Ryan, R. M. (1989). Parent styles associated with childrens self-regulation and competence in school. Journal of Educational Psychology, 81, 143-154.
Haladyna, T. M., Nolen, S. B., & Haas, N. S. (1991). Raising standardized achievement test scores and the origins of test score pollution. Educational Researcher, 20, 2-7.
Herman, J. L., Abedi, J., & Golan, S. (1994). Assessing the effects of standardized testing on schools. Educational and Psychological Measurement, 54, 471-482.
Herman, J. L., & Golan, S. (1993). The effects of standardized testing on teaching and schools. Educational Measurement: Issues and Practice, Winter, 20-42.
Hixson, B. K. (2000, January 25). How tests change a teacher. The New York Times.
Jeynes, W. H. (1999). The effects of children of divorce living with neither parent on the academic achievement of those children. Journal of Divorce & Remarriage, 30, 103-120.
Jones, K., & Whitford, B. L. (1997). Kentuckys conflicting reform principles: High-stakes school accountability and student performance assessment. Phi Delta Kappan, 70, 276-281.
Keith, T. Z., Keith, P. B., Quirk, K. J., Sperduto, J., Santillo, S., & Killings, S. (1998). Longitudinal effects of parent involvement on high school grades: Similarities and differences across gender and ethnic groups. Journal of School Psychology, 36, 335-363.
Lomax, R. G., West, M. M., Harmon, M. C., Viator, K. A., & Madaus, G. F. (1995). The impact of mandated standardized testing on minority students. Journal of Negro Education, 64, 171-185.
Lytton, H., & Pyryt, M. (1998). Predictors of achievement in basic skills: A Canadian effective schools study. Canadian Journal of Education, 23, 281-301.
Madaus, G. F. (1985). Test scores as administrative mechanisms in educational policy. Phi Delta Kappan, 66, 611-617.
McLean, R. (1997). Selected attitudinal factors related to students success in high school. The Alberta Journal of Educational Research, 43, 165-168.
Mathews, J., & Argetsinger, A. (June 2, 2000). Teachers say pressure to do well on tests is enormous. Washington Post, p. B1., A18.
Nikiforuk, A. (1997, April). When teachers sell kids short. Readers Digest, 119-120.
Paris, S. G., Lawton, T. A., Turner, J. C., & Roth, J. L. (1991). A developmental perspective on standardized achievement testing. Educational Researcher, 20, 12-20, 40.
Poole, P. (2000, May 31). Brave new schools: Clintons model school accused of cheating. WorldNetDaily (on-line). Available: www.worldnetdaily.com/ bluesky_exnews/ 2000531_xex_ clintons_mod.shtml.
Principles for fair student assessment practices for education in Canada. (1993). Edmonton, AB: Joint Advisory Committee. (Mailing address: Joint Advisory Committee, Centre for Research in Applied Measurement and Evaluation, 3-104 Education Building North, University of Alberta, Edmonton, AB, T6G 2G5).
Rotberg, I. C. (1995). Myths about test score comparisons. Science, 270, 1446-1448.
Salganik, L. H. (1985). Why testing reforms are so popular and how they are changing education. Phi Delta Kappan, 66, 607-610.
Scarborough, H. S., & Dobrick, W. (1994). On the efficacy of reading to preschoolers. Developmental Review, 14, 245-302.
Smith, M. L. (1991). Put to the test: The effects of external testing on teachers. Educational Researcher, 20, 8-11.
Whitaker, B. (1999, April 8). Prosecutor says indictment of Austin schools will help deter test tampering. The New York Times.
Wilgoren, J. (2000, February 25). Cheating on state wide tests is reported in Massachusetts. The New York Times.
Approved by the Board of Directors, Canadian Psychological Association, April, 2000, and by the Executive Committee, Canadian Association of School Psychologists, July, 2000.
Copyright © 2000
Canadian Psychological Association
Permission is granted to copy this document.
Canadian Psychological Association
Title: A Joint Position Statement by the Canadian Psychological Association and the Canadian Association of School Psychologists on the Canadian Press Coverage of the Province-wide Achievement Test Results
ISBN # 1896638630