New Jersey School Performance Reports: A Follow-up
Christopher Tienken
Bureaucrats at the New Jersey Department of Education (NJDOE) released a memo on April 11, 2013 to unveil the new, but in my opinion, not improved, performance reports for all schools for the 2011-2012 school year. Of course the NJDOE bureaucrats focused on student assessment data and comparisons to “peer” schools as the centerpiece of their new report. The new performance reports are now in their second year of existence and peer schools are still the focus of the reporting system.
Peer Schools
Peer schools comparisons? I was intrigued. Finally I thought that NJDOE bureaucrats were going to provide “clean” and useful comparisons of similar schools within the state. Needless to say I was disappointed by the comparisons provided. According to Bracey (2006), in order to compare things, they need to be comparable on the factors that influence the outcome one is comparing. Below are just a few methodological issues that should have been identified by NJDOE’s Chief Performance Officer/Assistant Commissioner of Data, Research, Evaluation and Reporting, and political science major, Bari Erlichson.
I use the peer group for Salem City High School as an example of how misleading and non-comparable some of the schools in the NJDOE created peer groups can be.
1. Free and Reduced Priced Lunch Differences: Free lunch and reduced lunch eligibility do not have the same effects on student achievement and should not be used as one category (Harwell & La Beau, 2011). It is the depth of poverty associated with being eligible for free lunch that depresses achievement. Reduced lunch eligibility has little negative influence on standardized tests scores.
In general, the higher the percentage of students in a school eligible for Free Lunch, the lower the percentage of students who score proficient or above on state tests. The NJDOE combines the categories of Free and Reduced Priced Lunches (FRPL) and that masks the real differences between schools in terms of poverty.
EXAMPLE #1: Things Are Not What They Seem with Poverty
The NJDOE performance report lists Salem City High School as having approximately 72% of its students eligible for Free and Reduced priced lunches. The NJDOE lists Harrison High School, a supposed “peer” of Salem as having approximately 85% of its students eligible. An initial, yet flawed conclusion might be, that the students in Salem are better off financially. However, when one looks at the Census data for those two towns it is apparent that Salem has a higher percentage of students living in poverty and higher percentages of children eligible for free lunch and that Harrison has more students eligible for reduced lunch.
Census data show that more than 55% of all households in Salem have incomes of less than $30,000 a year, whereas only 25% of households in Harrison live at those levels. Only 12% of families in Harrison live below the poverty line compared to 25% of families in Salem City. Twice as many families live below the poverty line in Salem City compared to families in one of its supposed peers districts, Harrison. The Census data suggest that Salem City has at least twice as many children eligible for free lunch than Harrison but you wouldn’t know that if you read the NJDOE peer group comparisons.
EXAMPLE #2: If I Could Choose My Team I Would Win Too
Perhaps the most egregious misuse of Free and Reduced Priced Lunch peer group comparisons comes from including charter schools and selective vocational school academies. NorthStar Academy in Newark is listed on the NJDOE peer group performance report as having over 83% of students eligible for FRPL. Unaware observers might interpret NorthStar as serving poorer children than those served by Salem City High School (approximately 72% FRPL). However, only about 52% of students in NorthStar qualify for Free lunch. Students eligible for free priced lunches simply do not attend charter schools at the same frequency they attend traditional public schools.
Similar issues occur at selective vocational school academies and magnet schools like Arts High School in Newark. The NJDOE reports that Arts has approximately 84% of students eligible for FRPL but only about 60% are eligible for free lunch. Furthermore, students are “Selected” to attend Arts High School and other magnet schools across the state. The administration at Salem City must accept all students and they do so enthusiastically. Schools that can select or deselect their students are not real public schools and should not be included in any comparisons of democratic public schools that accept all students regardless of prior achievement or poverty status.
2. The higher the percentage of students with special needs in a school the lower the percentage of students who score proficient or above. The range of percentages of students with special needs in the Salem City High School peer group is 0.9% to 29.8%. Salem City High School has approximately 25% of its student population categorized as special needs whereas Harrison had 14.7% and NorthStar had 7.8%. Not comparable.
3. The higher the percentage of students with English language learning needs (second language learners) in a school the lower the percentage of students scoring proficient or above on state tests. The range of percentages of students with English language learning needs in the Salem City High School peer group is from 0.1% to 27.3%. Not comparable.
In general, children from poverty with special education needs or English language learning needs are enrolled in charter schools, selective magnet schools, and selective vocational academies at lower percentages than in traditional, democratic, public schools. Approximately 1/3 of the schools in the Salem City High School peer group are not peers because they get to select their students or apply conditions that result in high levels of student attrition. They have lower percentages of students eligible for free priced lunches, lower percentages of students with special needs, and lower percentages of English language learners. Those schools are hardly comparable.
Peer Group Recommendations: First Steps
The NJDOE bureaucrats involved in determining peer groups should develop comparable peer groups. Schools that have entrance criteria of any type or use lotteries to select students should not be included in the peer groups with traditional, democratic public schools. They should form their own peer group of selective schools.
The percentage of students eligible for free lunch should be used along with community demographic indicators from the U.S. Census Bureau because socio-economic factors at the family and community level account for up to 65% of the variance in standardized test results. I have used Census data to predict the percentage of students scoring proficient or above at the district level on the NJASK tests at all grade levels with a high degree of accuracy. Socioeconomic factors matter greatly in terms of comparing school outputs and a variety of indicators could easily be incorporated into the peer group formulations.
For example, the NJDOE officials could develop a socioeconomic indicator basket made up of free lunch eligibility at the school level, the percentage of families living below the poverty line, and the percentage of lone parent households in the community. Schools comparable on those three demographic indicators could form the basis of the initial groupings. From there the groups should be further refined to include schools comparable on the socio-economic demographics and that have similar percentages of students with special needs (excluding students who receive speech/language services) and English language learners. Final comparison groups should have no less than 30 schools to allow for a basic level of statistical analysis.
College Readiness: Which Colleges?
In the original memo that unveiled the new performance report, NJDOE’s Chief Performance Officer/Assistant Commissioner of Data, Research, Evaluation and Reporting, Bari Erlichson (2013) stated:
While [sic] the evaluation of student outcome data is crucial for school improvement, we know that these data alone cannot capture the dozens of other essential elements of schools such as a positive school climate, participation in extracurricular programs and the development of non-cognitive skills. However, by focusing on college and career readiness and including meaningful comparisons for schools, we hope that these new reports will inform conversations at the local level about where schools are doing well and where they can continue to improve.
I was intrigued and fully support multiple measures of student achievement and school performance. So, what are the indicators of college and career readiness used by NJDOE bureaucrats?
The memo lists the following indicators for the high school levels:
• % of students participating in ACT, SAT, and PSAT
• % of students scoring above 1550 on the SAT, the benchmark score identified by the College Board associated with a high likelihood of college success.
• % of students taking at least one AP test in English, Math, Social Studies, or Science
• % of AP tests in English, Math, Social Studies, or Science with a score of 3 or greater
• % of students taking IB coursework
Apparently, the NJDOE bureaucrats are not aware of the almost 500 colleges and universities do not require the SAT or ACT. They also fail to recognize that neither the SAT or ACT were designed to be a measure of overall school quality. The bureaucrats seem not to realize that the benchmark SAT score of 1550 is the mean score used by the Top 20 public universities in the country. In essence, the NJDOE bureaucrats set the SAT benchmark to the one used by some of the most competitive public universities – thus students must be college-ready for the Top 20 colleges, not the remaining 4,400.
Algebra for All
Equally intriguing are the college and career readiness indicators at the elementary and middle school levels:
• % of students taking Algebra I
• % of students chronically absent (greater than 10% of the year)
It seems as if the wheels have officially come off the NJDOE Common Core bus with the use of % of students taking Algebra I in Grade 8 as a high-stakes indicator of school performance. Readers might recall that New Jersey bureaucrats were early adopters of the Common Core State Standards. Readers might also recall that all school districts in New Jersey were mandated to align all curriculum and resources to the Common Core State Standards in Mathematics and English Language Arts. Furthermore, astute readers know that the Common Core Mathematics Standards in Grade 8 do not call for the completion of traditional Algebra I in Grade 8.
Dr. Eric Milou from Rowan University parses this issue expertly in his earlier blog post on my site. Readers are strongly encouraged to see his analysis. Also, as detailed in Adding It Up: How Children Learn Mathematics, published by the National Academies Press (Kilpatrick, Swafford & Findell, 2001), formal Algebra I for all students in middle school is not recommended:
“In some countries by the end of eighth grade, all students have been studying algebra for several years, although not ordinarily in a separate course. “Algebra for all” is a worthwhile and attainable goal for middle school students. In the United States, however, some efforts to promote algebra for all have involved simply offering a standard first-year algebra course (algebra through quadratics) to everyone. We believe such efforts are virtually guaranteed to result in many students failing to develop proficiency in algebra, in part because the transition to algebra is so abrupt. Instead, a different curriculum is needed for algebra in middle school: “Algebra for all” is a worthwhile and attainable goal for middle school students. Teachers, researchers, and curriculum developers should explore ways to offer a middle school curriculum in which algebraic ideas are developed in a robust way and connected to the rest of mathematics.” (p. 420).
The NJDOE bureaucrats do not give credit to districts for having Grade 8 students in Algebra I unless it is a high school Algebra I course – exactly what the experts recommend NOT TO DO. Is anyone surprised that the NJDOE bureaucrats misinterpreted or disregarded research findings? As a long-time observer of NJDOE misuse of research, I am not. Parents, educators, and taxpayers should request a better system of reporting on the quality of New Jersey schools.
References
Bracey, G (2006). Reading educational research: How to avoid getting statistically snookered. NY: Heinemann.
Harwell, M., & LeBleau, B. (2010). Student eligibility for a free lunch as an SES measure in education research. Educational Researcher, 39, 120-131.
Kilpatrick, J., Swafford, J., & Findell, B., Editors; (2001). Adding It Up: How Children Learn Mathematics. National Academies Press.
