In 2014 I worked with an anonymous team of data specialists from Chicago Public Schools (CPS) to analyze test score data from public and charter schools to see which type of school fostered the most academic growth in its students. I shared our findings with the Chicago Sun-Times whose own data journalist, Art Golab, verified those findings and co-wrote an article about those findings with another Sun-Times reporter, Becky Schlikerman. One day after their article appeared, the Sun-Times published an on-line only Op-Ed I penned to elaborate on our findings. Less than a year after they were published, both of the above articles were scrubbed from the Sun-Times website. Here is a PDF link to the original print version. An image of that print version appears below.
Not long after those articles were published CPS officials in the “Office of Accountability” altered Charter school test score data. I exposed this in Power Concedes and it was later verified by the Sun-Times. Their tampering had the effect of improving the school rating for charters in gentrifying areas of Chicago (see report of tampering in the references section at the end of this article). It is for this reason that I’ve declined to do an analysis of test scores in subsequence years. CPS data post-2014 is simply not trustworthy.
When Mayor Rahm Emanuel recently heralded a small gain on the average Chicago Public Schools elementary “MAP” test results, I knew something wasn’t quite right. It wasn’t what he said; it was what he didn’t say. You see, this is the first year the MAP scores can provide a more decisive apples-to-apples comparison of charter schools and traditional public schools.
The result? Public school students learned far more in one year than charter school students did.
I reached this conclusion after analyzing this year’s MAP scores and I shared my results with several local reporters. Soon afterward, the Chicago Sun-Times conducted their own analysis for a story that was published on Sunday, Aug. 31. Their results differ from mine because they used a slightly different methodology, but the overall conclusions are the same: student growth in reading in neighborhood schools far outpaces growth in charters.
Measuring Student Learning
MAP stands for Measures of Academic Progress. Public and charter school students across the city took this national assessment in 2013. Each child got an individual score to serve as a baseline. By calculating the difference between that score and the 2014 score, CPS can determine the amount of learning growth each child attained in the year between the exams.
Until now, schools were judged on student attainment scores, not student growth. This is important because — like magnet schools — charter schools lean heavily on their ability to enroll students who are more likely to have higher attainment than their neighborhood peers by virtue of the degree of parent involvement needed to enter a child into a charter school lottery. Chicago’s charter schools also expel students at more than 12 times the rate of our public schools, which calls into question their own confidence in their ability to effectively teach the most difficult-to-reach children. When you consider those factors, the attainment of charter school students could be more a result of their admissions and expulsion policies than their teaching.
This is where the MAP assessment comes in. The MAP is designed to measure teaching and learning. In fact, CPS trusts it so much that it uses the results to determine teacher and principal evaluation ratings. It’s also used to rate schools on CPS’s five-level rating system.
If CPS can use MAP growth results so broadly to rate teachers, principals, and schools, one would expect CPS to use those same results to rate its school reform strategy, which is dominated by the proliferation of privatized charter and turnaround schools, where a private operator replaces all or nearly all of a school’s staff. How did Emanuel’s reform schools do? What kind of learning growth did they foster? Why didn’t Mr. Emanuel say anything about it? Surely he knew.
The Research and Analysis
So I downloaded the publicly available MAP results and conducted a preliminary analysis. For the sake of consistency I am deferring to the Sun-Times results, which were similar to mine. The MAP report lists the “growth percentile” assigned to each school based on student results. If a school gets a growth percentile of 99, then the average growth of the students in that school is greater than the average growth of 99 percent of schools in the United States that took the MAP assessment.
In terms of assessing the effectiveness of charter schools, I believe the most accurate comparison is to public magnet schools since both charters and magnets have lottery admissions processes that increase the likelihood of enrolling students with involved parents. In essence, charters are privately run magnet schools and therefore should be measured against publicly run magnet schools. I believe that turnaround schools should be compared to neighborhood schools since they both must accept students within their attendance boundaries. Using the Sun-Times results, the comparisons are as follows:
* The most dramatic performance gaps are in reading, where the public magnet school growth percentile is 83, while the charter score is 48.
* The public neighborhood percentile is at 75, while turnarounds are at 51.
* Although neighborhood schools must enroll any student in their attendance boundary, their students’ reading growth percentile is 27 points higher than that of lottery-driven charters schools. Neighborhood schools are at 75 and charters are at 48.
* In math, the public magnet school growth percentile is 67, while the lottery-driven charter schools are at 49.5 — over 17 points lower.
* The neighborhood school growth percentile is at 55 while the turnaround school percentile is at 43 — 12 points lower.
* Even with their admissions limitations, public neighborhood schools outperformed the growth in lottery-driven charter schools by more than five percentile points, with neighborhood and charter schools at 54.9 and 49.5 respectively.
A simple look at a list of the schools reveals even more. Of the 490 Chicago schools for which elementary grade MAP data was available, 60 of those schools are charter (12 percent), 24 are turnaround (5 percent), and 406 (83 percent) are traditional public schools. When sorted by growth percentile rank, I found the following:
* Although charters and turnarounds make up 17 percent of district schools, they account for none of the 45 schools with the highest growth percentiles.
* Of the 30 lowest performing schools in CPS more than half are charters or turnarounds.
* Of the 10 lowest-performing schools in CPS, six are charters or turnarounds.
* Nearly nine out of 10 charter/turnaround schools are in the bottom half of CPS performance.
It’s comparing apples to oranges. The charter and turnaround schools in question do not compare to the top 45 CPS public schools. Not only because of achievement, but mostly because of demographics and geographic location.
Looking at these results in local neighborhoods provides further insight. In order to understand this, one must have an accurate understanding of student academic attainment vs. student academic growth. The following scenario is illustrative:
Teacher A’s students grow far more than Teacher B’s, but their attainment is still behind Teacher B’s students because they started behind. Teacher A’s students started four years behind Teacher B’s, but at the end of the year they were only two years behind. If the teachers are judged on the attainment of their students, then Teacher B will come out on top even though her students only grew a half year. This is why looking at attainment scores can be a profoundly deceptive way to judge the quality of teaching. A more accurate way of looking at teaching and learning is to look at student academic growth. Although many growth measures have problems of their own, they are a far better reflection of teaching and learning than attainment scores.
With that in mind, let’s look at attainment vs. growth in specific Chicago neighborhoods.
As you can see, students from one of the two (highlighted) charter schools represented in this area of the northwest side of Chicago have relatively high attainment (#5 of 15 schools). As was the case with “Teacher B” the attainment of the students is not an accurate measure of the quality of teaching and learning in a school. Charters have registration procedures that make it more likely they’ll enroll higher attaining students in the first place, and when they do enroll lower attaining students, Chicago’s charters are notorious for expelling struggling students and/or pressuring their families to enroll them elsewhere. As a result, we have to look at a more accurate measure–academic growth.
So while the top chart shows CICS Irving Park at #5 in attainment, it is dead last in the amount of academic growth it fosters in the students it enrolls. This pattern repeats itself all across Chicago’s neighborhoods.
Once again, on the opposite side of the city, we see the same forces at work. The top chart shows that the three Charter schools in West Pullman and Altgeld Gardens manage to enroll students whose attainment is comparable to most neighborhood schools, but then all three of those charters are dead last when it comes to measures of what students actually learn when they attend those charters (growth). It is important to note that parents made the “choice” to take their children out of schools like Higgins (99th percentile), Brown (99th) and Dubois (80th) and send them to charters like Prairie (6th), Hawkins (3rd) and Lloyd Bond (1st). If parents choose to send their children to schools where they don’t grow academically, then the “parent trigger” theory embedded in the choice model is completely disproven.
Once again, in Chatham, the pattern of charters school students with average attainment but low growth repeats itself, disproving the assumptions behind the choice model.
CPS testing and accountability officials told me their numbers looked similar to mine and that any minor differences may have been the result of the inclusion of one or two schools not included in the data available at the time of my analysis. This led to another striking revelation. Eight of the city’s charter schools — including five Learn Charter Schools — had no MAP growth data at all. When I asked how this was possible I was told these charters had not “opted in” to the MAP assessment. You read that correctly; CPS allows some charter schools not to participate in the assessment used to hold regular public schools accountable.
The 50th percentile represents the “average” for U.S. schools. The reading growth percentile scores of 83 and 75 for students in Chicago’s public magnet and neighborhood schools stands in stark contrast to the often-promoted picture of traditional Chicago public schools as “failing.” On the contrary, the 51st and 48th respective growth percentiles of turnarounds and charters clearly indicate that it is these reforms that are failing Chicago’s students. There may be a few exceptions, but exceptions don’t create good schools systems; critical mass does. Our public schools have developed this critical mass while charter schools have fallen short.
This situation sets up an inexcusably dire situation when considered in the context of the racial achievement gap. As large numbers of African-American and Hispanic students are funneled into the low-growth charter/turnaround system, the high-growth public system is becoming increasingly Caucasian and Asian. The students on the low end of the achievement gap — the students who need the most growth — are being fed to a system that produces the least. In December, the Sun-Times reported that the achievement gap between white and black students was widening. It now appears we have identified a cause.
In the face of these results, the mayor’s next press conference on schools should be much different from his last. He should announce that CPS will cease its effort to divert funding from public neighborhood schools into his failed charter experiment. An immediate surge of investment in public neighborhood schools should follow. He should also announce an immediate publicity campaign to inform parents who made the charter “choice” of the learning growth disparity between these different types of schools so those parents can then make a more informed choice about where to send their children. Unfortunately, many of the schools in those parents’ neighborhoods have been shut down. It is a tragic irony that a so-called “choice” system has left thousands of families with no choice at all.
In the past, when public school advocates have mentioned the difficulties of teaching in schools in low-income minority neighborhoods, charter and “choice” advocates have had a “no excuses” response. “Hold the public schools accountable!” has been the battle cry. Will the mayor now hold his charter schools accountable? Let’s hope Mr. Emanuel remains consistent with that “no excuses” mantra now that his own reforms have failed.
SPREAD THE WORD
Fight Charter Expansion!
If you belong to an organization or group and would like to arrange a live presentation and explanation of the information and insights contained in this post, contact me at the email address below.
LaRaviere, Troy (December 10,2014). Chicago School Officials Alter Charter School Test Score Data.