In 2014 I worked with an anonymous team of data specialists from Chicago Public Schools (CPS) to analyze test score data from public and charter schools to see which type of school fostered the most academic growth in its students. I shared our findings with the Chicago Sun-Times whose own data journalist, Art Golab, verified those findings and co-wrote an article about those findings with another Sun-Times reporter, Becky Schlikerman. One day after their article appeared, the Sun-Times published an on-line only Op-Ed I penned to elaborate on our findings. Less than a year after they were published, both of the above articles were scrubbed from the Sun-Times website. Here is a PDF link to the original print version. An image of that print version appears below.
Not long after those articles were published CPS officials in the “Office of Accountability” altered Charter school test score data. I exposed this in Power Concedes and it was later verified by the Sun-Times. Their tampering had the effect of improving the school rating for charters in gentrifying areas of Chicago (see report of tampering in the references section at the end of this article). It is for this reason that I’ve declined to do an analysis of test scores in subsequence years. CPS data post-2014 is simply not trustworthy.
When Mayor Rahm Emanuel recently heralded a small gain on the average Chicago Public Schools elementary “MAP” test results, I knew something wasn’t quite right. It wasn’t what he said; it was what he didn’t say. You see, this is the first year the MAP scores can provide a more decisive apples-to-apples comparison of charter schools and traditional public schools.
The result? Public school students learned far more in one year than charter school students did.
I reached this conclusion after analyzing this year’s MAP scores and I shared my results with several local reporters. Soon afterward, the Chicago Sun-Times conducted their own analysis for a story that was published on Sunday, Aug. 31. Their results differ from mine because they used a slightly different methodology, but the overall conclusions are the same: student growth in reading in neighborhood schools far outpaces growth in charters.
Measuring Student Learning
MAP stands for Measures of Academic Progress. Public and charter school students across the city took this national assessment in 2013. Each child got an individual score to serve as a baseline. By calculating the difference between that score and the 2014 score, CPS can determine the amount of learning growth each child attained in the year between the exams.
Until now, schools were judged on student attainment scores, not student growth. This is important because — like magnet schools — charter schools lean heavily on their ability to enroll students who are more likely to have higher attainment than their neighborhood peers by virtue of the degree of parent involvement needed to enter a child into a charter school lottery. Chicago’s charter schools also expel students at more than 12 times the rate of our public schools, which calls into question their own confidence in their ability to effectively teach the most difficult-to-reach children. When you consider those factors, the attainment of charter school students could be more a result of their admissions and expulsion policies than their teaching.
This is where the MAP assessment comes in. The MAP is designed to measure teaching and learning. In fact, CPS trusts it so much that it uses the results to determine teacher and principal evaluation ratings. It’s also used to rate schools on CPS’s five-level rating system.
If CPS can use MAP growth results so broadly to rate teachers, principals, and schools, one would expect CPS to use those same results to rate its school reform strategy, which is dominated by the proliferation of privatized charter and turnaround schools, where a private operator replaces all or nearly all of a school’s staff. How did Emanuel’s reform schools do? What kind of learning growth did they foster? Why didn’t Mr. Emanuel say anything about it? Surely he knew.
The Research and Analysis
So I downloaded the publicly available MAP results and conducted a preliminary analysis. For the sake of consistency I am deferring to the Sun-Times results, which were similar to mine. The MAP report lists the “growth percentile” assigned to each school based on student results. If a school gets a growth percentile of 99, then the average growth of the students in that school is greater than the average growth of 99 percent of schools in the United States that took the MAP assessment.
In terms of assessing the effectiveness of charter schools, I believe the most accurate comparison is to public magnet schools since both charters and magnets have lottery admissions processes that increase the likelihood of enrolling students with involved parents. In essence, charters are privately run magnet schools and therefore should be measured against publicly run magnet schools. I believe that turnaround schools should be compared to neighborhood schools since they both must accept students within their attendance boundaries. Using the Sun-Times results, the comparisons are as follows:
* The most dramatic performance gaps are in reading, where the public magnet school growth percentile is 83, while the charter score is 48.
* The public neighborhood percentile is at 75, while turnarounds are at 51.
* Although neighborhood schools must enroll any student in their attendance boundary, their students’ reading growth percentile is 27 points higher than that of lottery-driven charters schools. Neighborhood schools are at 75 and charters are at 48.
* In math, the public magnet school growth percentile is 67, while the lottery-driven charter schools are at 49.5 — over 17 points lower.
* The neighborhood school growth percentile is at 55 while the turnaround school percentile is at 43 — 12 points lower.
* Even with their admissions limitations, public neighborhood schools outperformed the growth in lottery-driven charter schools by more than five percentile points, with neighborhood and charter schools at 54.9 and 49.5 respectively.
A simple look at a list of the schools reveals even more. Of the 490 Chicago schools for which elementary grade MAP data was available, 60 of those schools are charter (12 percent), 24 are turnaround (5 percent), and 406 (83 percent) are traditional public schools. When sorted by growth percentile rank, I found the following:
* Although charters and turnarounds make up 17 percent of district schools, they account for none of the 45 schools with the highest growth percentiles.
* Of the 30 lowest performing schools in CPS more than half are charters or turnarounds.
* Of the 10 lowest-performing schools in CPS, six are charters or turnarounds.
* Nearly nine out of 10 charter/turnaround schools are in the bottom half of CPS performance.
It’s comparing apples to oranges. The charter and turnaround schools in question do not compare to the top 45 CPS public schools. Not only because of achievement, but mostly because of demographics and geographic location.
Looking at these results in local neighborhoods provides further insight. In order to understand this, one must have an accurate understanding of student academic attainment vs. student academic growth. The following scenario is illustrative:
Teacher A’s students grow far more than Teacher B’s, but their attainment is still behind Teacher B’s students because they started behind. Teacher A’s students started four years behind Teacher B’s, but at the end of the year they were only two years behind. If the teachers are judged on the attainment of their students, then Teacher B will come out on top even though her students only grew a half year. This is why looking at attainment scores can be a profoundly deceptive way to judge the quality of teaching. A more accurate way of looking at teaching and learning is to look at student academic growth. Although many growth measures have problems of their own, they are a far better reflection of teaching and learning than attainment scores.
With that in mind, let’s look at attainment vs. growth in specific Chicago neighborhoods.
As you can see, students from one of the two (highlighted) charter schools represented in this area of the northwest side of Chicago have relatively high attainment (#5 of 15 schools). As was the case with “Teacher B” the attainment of the students is not an accurate measure of the quality of teaching and learning in a school. Charters have registration procedures that make it more likely they’ll enroll higher attaining students in the first place, and when they do enroll lower attaining students, Chicago’s charters are notorious for expelling struggling students and/or pressuring their families to enroll them elsewhere. As a result, we have to look at a more accurate measure–academic growth.
So while the top chart shows CICS Irving Park at #5 in attainment, it is dead last in the amount of academic growth it fosters in the students it enrolls. This pattern repeats itself all across Chicago’s neighborhoods.
Once again, on the opposite side of the city, we see the same forces at work. The top chart shows that the three Charter schools in West Pullman and Altgeld Gardens manage to enroll students whose attainment is comparable to most neighborhood schools, but then all three of those charters are dead last when it comes to measures of what students actually learn when they attend those charters (growth). It is important to note that parents made the “choice” to take their children out of schools like Higgins (99th percentile), Brown (99th) and Dubois (80th) and send them to charters like Prairie (6th), Hawkins (3rd) and Lloyd Bond (1st). If parents choose to send their children to schools where they don’t grow academically, then the “parent trigger” theory embedded in the choice model is completely disproven.
Once again, in Chatham, the pattern of charters school students with average attainment but low growth repeats itself, disproving the assumptions behind the choice model.
CPS testing and accountability officials told me their numbers looked similar to mine and that any minor differences may have been the result of the inclusion of one or two schools not included in the data available at the time of my analysis. This led to another striking revelation. Eight of the city’s charter schools — including five Learn Charter Schools — had no MAP growth data at all. When I asked how this was possible I was told these charters had not “opted in” to the MAP assessment. You read that correctly; CPS allows some charter schools not to participate in the assessment used to hold regular public schools accountable.
The 50th percentile represents the “average” for U.S. schools. The reading growth percentile scores of 83 and 75 for students in Chicago’s public magnet and neighborhood schools stands in stark contrast to the often-promoted picture of traditional Chicago public schools as “failing.” On the contrary, the 51st and 48th respective growth percentiles of turnarounds and charters clearly indicate that it is these reforms that are failing Chicago’s students. There may be a few exceptions, but exceptions don’t create good schools systems; critical mass does. Our public schools have developed this critical mass while charter schools have fallen short.
This situation sets up an inexcusably dire situation when considered in the context of the racial achievement gap. As large numbers of African-American and Hispanic students are funneled into the low-growth charter/turnaround system, the high-growth public system is becoming increasingly Caucasian and Asian. The students on the low end of the achievement gap — the students who need the most growth — are being fed to a system that produces the least. In December, the Sun-Times reported that the achievement gap between white and black students was widening. It now appears we have identified a cause.
In the face of these results, the mayor’s next press conference on schools should be much different from his last. He should announce that CPS will cease its effort to divert funding from public neighborhood schools into his failed charter experiment. An immediate surge of investment in public neighborhood schools should follow. He should also announce an immediate publicity campaign to inform parents who made the charter “choice” of the learning growth disparity between these different types of schools so those parents can then make a more informed choice about where to send their children. Unfortunately, many of the schools in those parents’ neighborhoods have been shut down. It is a tragic irony that a so-called “choice” system has left thousands of families with no choice at all.
In the past, when public school advocates have mentioned the difficulties of teaching in schools in low-income minority neighborhoods, charter and “choice” advocates have had a “no excuses” response. “Hold the public schools accountable!” has been the battle cry. Will the mayor now hold his charter schools accountable? Let’s hope Mr. Emanuel remains consistent with that “no excuses” mantra now that his own reforms have failed.
SPREAD THE WORD
Fight Charter Expansion!
If you belong to an organization or group and would like to arrange a live presentation and explanation of the information and insights contained in this post, contact me at the email address below.
LaRaviere, Troy (December 10,2014). Chicago School Officials Alter Charter School Test Score Data.
24 thoughts on “Drop Charter Reform Strategy: Chicago’s Public School Students Outlearn Students from Charters”
Reblogged this on David R. Taylor-Thoughts on Education.
Support Troy in his efforts to expose the truth by posting and talking of his findings as much as possible!!
This article is amazing, thank you so much for the in-depth analysis, and (more importantly, b/c not all of us are analysis geeks 😉 ) explaining the subtle but incredibly important differences b/t measurements of attainment, and growth.
Thank you for this great article.
I have no doubt that similar results can be found elsewhere.
The article is here: http://chicago.suntimes.com/politics/drop-cps-reform-strategy-neighborhood-schools-best-charters/
It’s just the first couple paragraphs. The “continue reading” link takes you to a “page not found” landing.
Troy, have you been able to do a similar comparison for high schools?
Comparable growth data does not exist. They have something they tout as equivalent but it does not control for beginning test score. If we were able to look act EPAS growth in a way that controls for the beginning test score a student has and compares them only with other students that started at that same test score then we would have a reliable comparison. Right now they just look at total growth which favors schools that start off with students who began with higher test scores because they grow more due to to lack of a curriculum in high schools tailored to low scoring student who are still struggling to master basic elementary concepts. They tend not to grow because high schools don’t have the kind of curriculum that’s designed for their level of functioning.
It’s odd that this Article would be published with Rahm playing the tough guy with Aramark (http://chicago.suntimes.com/politics/rahm-to-aramark-clean-up-the-schools-or-clean-out-your-desks/), but right now, aramark has positions posted for engineers for the chicago public schools when no contract has been signed or released publicly, and local 143 engineers think they are bargaining for a four year contract. Talk about things that make you go “hmm…” Did rahm ever reveal what happened when they did a side by side comparison between public sector workers and private companies for garbage collection?? Really curious about this one.
Thank you. journalist is a possible side gig
At the last Board of Education meeting, the Board approved a dozen charter expansion and new charter proposals with no reference to any of the facts, such as those you are reporting. Who will be reporting these facts at the May 25 Board meeting — even if the Board is going to continue to ignore the facts?
Rate of growth is clearly the metric, thank you. As a Special Education teacher it’s been vital to have a student understand they may not hit a particular cut score at a particular time, but maybe (often!) they grew two years in one using these measures and THAT is far more impressive. Opens the door for GRIT, Growth Mindset and a host of other critical transition skills.
You’re growing your brand well, and speak to the truth nicely. I hope you are considering a run for Mayor, we need a champion like yourself.
Thanks for sharing this data. I’m not an advocate for charter schools, but there is a flaw in this analysis. You are comparing apples to oranges. The charter and turnaround schools in question do not compare to the top 45 CPS public schools. Not only because of achievement, but mostly because of demographics and geographic location. Not a valid comparison.
Your comment seems to indicate you looked at the chart without reading the article. If it were comparing overall student academic attainment then you’d have a point. But it’s comparing student academic GROWTH so it’s apples to apples. In addition, you have made a false assumption about the demographics of the top 45 schools for academic growth. There are plenty of schools with challenging demographics in the list of the top 45, like Ariel, Brown, Caldwell, Cullen, Doolittle, Dunne, Fernwood, Foster Park, Hefferan, Higgins, Kershaw, Madison, Nash, Powell, Shmid, Sherwood, Ward, and Warren; all which are over 95% African American and majority low-income; yet each is in the 99th percentile for the amount of academic growth they foster in their students. In addition even if I took the top 45 schools out of the equation that still doesn’t explain why charters are over-represented by a landslide in the worst 45 schools considering they only make up a small percentage of CPS. Most regular public schools in low income communities have far more challenging demographics than charter schools do by virtue of the fact that they must accept all students rather than only those students whose parents enter them into a the charter/magnet lottery. Yet charters are still embarrassingly overrepresented among the schools with the lowest student academic growth. The picture is clear: Charter schools are an abysmal failure, and the sooner we come to terms with this fact, the better course we can plan for the future of our school system and the children it serves.
I’ve never liked charter schools. Schools for profit are no better then prisons for profit. The bottom line is more important then the education of the student. Wrong focus. The article is enlightening. Taxes pay for our schools. Replace the charters with CPS. Parents and students deserve no less.
Great article. You speak the truth. We need to all get behind Mr. Troy Laraviere.
Thanks for this great analysis. It seems to me like the obvious follow-up question is, “Why?” What are the qualitative reasons for these statistical trends, and how can we learn from whatever mistakes are being made right now in failing charters?
In essence, you get what you pay for.
You include the AUSL turnaround schools in the “charter list.” Those are CPS schools, run and funded by CPS, abiding by all CPS policies, district-wide testing, the Student Code of Conduct, etc. AUSL provides instructional coaches (rather than ISLs) and some additional PD opportunities, but they are nothing like Charter schools. Do your homework on the AUSL model and network.
1. I appears that you didn’t read the article. If you actually read it instead of just looking at a list, you would know I included AUSL turnaround schools because they–along with charters–are a part of this administration’s education agenda. I asked the question in paragraph 9, “How did Emanuel’s reform schools do?” The point is that both of this administrations so-called solutions–Charters and Turnarounds–are only making things worse.
2. If you had done your “homework” on me, you’d know that I am more than familiar with the AUSL model because I was once an AUSL administrator, having led one of its more successful turnarounds (with Alice Henry). When we took over Johnson Elementary School in 2009 40% of students were meeting state standards. Two years later–with the same population of students–that number was over 60%. After I left, CPS threw all of that progress into turmoil not once but twice, making Johnson a so-called “welcoming” school in two successive years when it closed nearby Lathrop in 2012, and Pope in 2013. As I stated in the article you didn’t appear to read, there are some charters and turnarounds that are exceptions to the overall trend of poor student growth, but exceptions don’t make good school systems: critical mass does. Charters and Turnarounds have not created that critical mass, but traditional public schools have. Lastly, although AUSL schools are indeed still public neighborhood schools, they–like charters–are run by a private operator.
It looks like someone certainly needs to do her homework, but that someone isn’t me.
Troy- can you take your energy and enthusiasm into the education funding space (and maybe you have) and fight that fight right now which is so critical?