New York City's Charter Schools: What the Research Shows

This report evaluates the current state of research on New York City charter schools. Overall, their effect on student performance is unambiguously positive. But the research is more dated and limited in scope than proponents and critics of charters appreciate.

• Students who attend a New York City charter instead of a traditional public school do much better on math tests and better, but by a smaller amount, on English language arts (ELA) tests. These positive effects appear to have remained similar over time, even as the number of charters and students has expanded. However, the studies that employ the strongest potential research design are dated or targeted only at a few highly effective schools that do not represent the full charter sector today.
• New York City charter schools are not equally effective. About half appear to be more effective than the traditional public school that students would have otherwise attended. A small percentage of charters appear to have negative effects on student test scores.
• Differences in resources do not explain differences in effectiveness between charter schools and traditional public schools or between charters in New York City.
• There is no evidence that New York City charter schools systematically push out low-performing students. In fact, low-performing students are less likely to exit charter schools than they are to exit traditional public schools, especially after accounting for differences in their demographic characteristics.
• Competition from New York City charter schools has either no effect, or a positive effect, on the performance of students in the nearby traditional public schools.

READ FULL REPORT

______________________

Marcus Winters is a senior fellow at the Manhattan Institute and an associate professor at Boston University. Follow him on Twitter here. 

This report evaluates the current state of research on New York City charter schools. Overall, their effect on student performance is unambiguously positive. But the research is more dated and limited in scope than proponents and critics of charters appreciate.

While the research on charter schools across the U.S. is growing, this paper focuses exclusively on studies that evaluate one or more aspects of New York City charter schools. This focus is important because the characteristics and effects of charter schools vary from city to city and, indeed, from charter to charter.

If you ask five people what the academic research says about New York City charter schools, you might get seven different answers. Indeed, the empirical evidence is now so broad and long-standing that it is worth stepping back to reflect on where it stands.

This report reviews existing research, which shows that New York City charter schools have an unambiguously positive effect on student performance. But the research is more dated and limited in scope than is commonly appreciated.

I focus exclusively on the results from studies that specifically evaluate one or more aspects of New York City’s charter sector because the characteristics and impacts of charter schools vary dramatically by locality.[1] In New York and elsewhere, conversations about the impact of charter schools are often contaminated by discussions of the national charter school literature. That literature provides a broad understanding of the charter school movement and offers insights into why charters appear to be more effective in some localities than in others. But in cases like New York City, where broadly informative local research is available, the policy conversation should focus explicitly on the local body of research. (In cases where research on non–New York charter school sectors might inform this report, I include a brief description in the endnotes.)

I have done my best to collect all of the relevant academic literature on New York City charter schools in this review. (See the Appendix for further readings.) I do not impose a firm cutoff date to include a paper, but I do exclude early studies that are clearly no longer relevant.[2] In addition to employing my own knowledge of the literature, I searched on Google Scholar, browsed the websites of organizations—supporters as well as opponents—that frequently write about the city’s charter sector in the popular press,[3] and collected further relevant papers that were cited in the initial papers that I identified.

I do not restrict the review to published academic work. Peer review is an imperfect signal of a study’s quality. In some cases, I reference published critiques of a particular study, but I do not treat such reviews as evaluations. I organize this review as a series of questions and answers—a Q&A format seemed best for addressing the most pressing claims and controversies among the public, as well as among the scholarly community.

New York City charter schools post substantially higher test scores, on average, than do the city’s traditional public schools. Alas, the aggregate test scores of charter and traditional public schools are not directly comparable. Differences in the performance of charter and traditional public school students might reflect differences in the quality of schooling, student differences, or both. Measuring the effect of attending a charter school on student performance requires rigorous statistical analysis.

Seven studies estimate the causal effect of attending a New York City charter school on student performance—i.e., they attempt to directly answer the question of whether the same charter school student would have performed differently had he or she attended a traditional public school.[4] These studies use one of two research designs, both of which have benefits and limitations that should be considered when interpreting their results.

Five use a randomized field trial (RFT) design. This design takes advantage of the fact that oversubscribed charter schools are required to use a random lottery to offer students the opportunity to enroll. The researchers compare the later outcomes of students who were randomly offered a charter school seat with those of students who also applied but were randomly denied the opportunity to enroll.

Because winning the charter lottery is not related to any other student characteristic, any differences in the later outcomes of lottery winners and losers are either random or are directly due to the opportunity to enroll in a charter school. Researchers then statistically adjust the estimated effect of being offered a charter seat (the “reduced form” or “intention-to-treat” estimate) to account for the fact that not all students who are offered a charter school seat actually attend the school (the treatment-on-the-treated estimate). Because the RFT design accounts for all pretreatment differences between the treatment and control groups, it is widely considered to be the gold standard for research designs in the social sciences.

From a policy perspective, RFT studies produce limited information because their results strictly apply only to those represented in the lottery pools. Thus, if students who apply to charter schools are systematically different from those who do not apply to them, then the estimates from an RFT study likely would not apply to non-applicants.[5] Many RFT studies of New York City charter schools also focus exclusively on one or several charter operators; and the schools included are typically among the highest-performing. Such studies reveal a great deal about the effect of attending a particular charter school, but they do not offer a complete picture of the effectiveness of New York’s charter school sector.

The second research design, “matching,” uses a computer algorithm to compare the performance of each student attending a charter school with a similar student or students attending a traditional public school. The Center for Research on Educational Outcomes (CREDO) at Stanford University has used this design to estimate the effects of charter schools on student performance in New York City and elsewhere (two of its matching studies are discussed below). Unlike the RFT design, matching allows the researcher to include all charter schools, regardless of whether they are oversubscribed or participate in the evaluation.

Matching requires stronger assumptions than RFT to produce estimates of the causal impact of attending a charter school. RFT studies control for differences between applicants who are offered and not offered a charter seat, whether or not the difference is observed in the researcher’s data set. Matching studies match students only according to characteristics that are observed in the data set, even though students attending charters may differ from those attending traditional public schools in unobserved ways, such as the level of parental involvement.

Figures 1–2 plot the main estimates for the impact of attending a New York City charter school on student math and ELA scores, as reported in the seven studies.[6] The five RFT studies (blue dots) list the school included in the analysis, as well as the studies’ authors. The two matching studies (yellow dots) are CREDO (2013) and CREDO (2017); their estimates hold for all charter schools operating in a given year in New York City.

Figures 1–2 show that the estimated effect of attending a New York City charter school is positive in all but one case; 23 of 28 estimates are significantly positive. Attending a New York City charter appears to have a larger effect in math than in ELA. The magnitude of the positive effect in math is substantial, especially given that the estimate covers only a year of charter school attendance. The smallest estimate suggests that four years of attending a charter school would lead to a 0.28 standard deviation increase in student math scores (0.07 x 4 years). The estimates in math are also quite stable over time and across studies.

Meanwhile, the estimated effects on student performance in ELA are more modest and less consistent. However, the evidence suggests a significant positive effect in ELA that would be meaningful after accumulating over time.

As noted, RFT and matching studies have strengths and weaknesses. Three of the RFT studies evaluate only a single, high-performing charter network (Promise Academy, Democracy Prep, or Success Academy). A fourth uses information from a more expansive group of 29 schools. Hoxby and Murarka (2009), the fifth RFT study and the only one to include all of the city’s then-operating charter schools, found a significant positive impact in math and ELA.

Though Hoxby and Murarka remain highly influential in the policy discussion, their results are quite dated. New York City’s charter sector has grown dramatically (Figure 3) since 2006, the final year of study in Hoxby and Murarka. In 2006, only 47 charter schools operated in New York, compared with 227 today. Because of this big expansion, the effect of attending a charter school may have changed over time.

If, say, the city successfully closed ineffective charters and encouraged the opening of only highly effective ones, the impact of attending a charter school would tend to increase over time. On the other hand, the quality of charter schools might diminish as the sector expands and digs deeper into the labor pool for administrators and teachers, or if the additional students who enroll are, for some reason, less responsive to charter schooling than the average student who enrolled previously.

Compare matching and RFT still further. The fact that the RFT studies often include only a small minority of charter schools likely explains why they tend to produce substantially larger effect size. Meanwhile, each matching estimate produces a statistically significant result; but four of 10 RFT estimates are statistically insignificant. The reason: the RFT estimates are measured imprecisely, partly because they utilize far fewer schools and students than do the matching estimates.

Are the matching estimates reasonable? As of 2006, the estimated impact of attending a charter school using RFT (0.06 standard deviations in ELA and 0.09 standard deviations in math) was similar to the estimated impact of attending a charter school in 2007 using matching (0.10 standard deviations in ELA and 0.12 standard deviations in math). The fact that these approaches—applied at about the same time, using nearly the same sample of schools—produced relatively similar results suggests that matching is capable of producing causal estimates for the effect of attending a New York City charter school. Indeed, the matching estimates have been relatively consistent over time, especially in math. This suggests that the average effectiveness of the city’s charter sector has not diminished, even as it has expanded.[15]

The empirical research described above strongly suggests that, on average, students attending a charter school score higher in math and ELA than they would have had they attended a traditional public school. However, charters are not monolithic: they are separate, independently operated, schools. The type of schooling offered by charters varies substantially in New York, too. Charters also have access to varying levels of financial resources, from public and private sources. Thus, while charter schools might be effective on average, not all New York City charter schools are equally effective.

The potential for variation in the effectiveness of charter schools is visible in Figures 1–2. As discussed, the RFT studies tend to produce larger estimated effects than do matching studies. Some of that difference could result from differences in research methodologies. But it could also be due to the fact that the RFT studies focus on a set of highly effective schools, which, by definition, implies variation in charter school quality. Dobbie and Fryer (2013) is the only RFT study to produce an estimated charter school effect (in math) worse than those produced by matching estimates in a given year. It is also the only RFT study since Hoxby and Murarka (2009) to include a broad set of charter schools rather than only those in a single network.

The matching studies offer direct evidence that the quality of New York’s charters varies, too. CREDO (2017), which disaggregates results by school, finds that 47.7% of the city’s charter schools produced significant gains in math when compared with traditional public schools, 35.3% scored equally well, and 17% scored worse. It finds similar results for ELA.

CREDO (2013) and CREDO (2017) also find variation in the quality of charter school networks. Among networks operating at least four charters in New York, the effect of attending a network school ranges from –0.06 to 0.24 standard deviations in ELA and (an insignificant) 0.00 to 0.42 standard deviations in math.

Some studies attempt to identify characteristics of successful charter schools by measuring each school’s impact on student performance, using RFT or matching, and then they correlate the measured effect of each school with one or more characteristics of the school. Keep in mind that the estimated charter school effect can be considered to be causal, but the associations between effectiveness and observed characteristics of the school cannot be considered to be causal. (The attributes of these schools are largely fixed and are not randomly assigned, which makes it impossible to distinguish between their effects and the effects of the schools’ unobserved attributes.) These studies can therefore identify characteristics that tend to be found in the more successful schools, but they cannot prove that the same (positive) results would occur if adopted in other schools.

CREDO (2017) finds that charter schools operated by charter management organizations (CMOs)—organizations that manage networks of several schools under a common leadership and philosophy—tend to have substantially larger positive effects on their students than do charters operated by non-CMOs. (For charter schools operated by non-CMOs, the study still finds a significantly positive effect in math but an insignificant effect in ELA.)

The RFT studies that utilize several different charter schools also attempt to find correlations between characteristics and measured school quality. Hoxby and Murarka (2009), for example, find that longer school years are associated with superior quality. But they do not find other characteristics that are robustly associated with measured school effectiveness.

Dobbie and Fryer (2013) present the most thorough analysis of the underlying factors associated with charter school quality to date. They supplement reported information for each school with detailed survey data and, in some cases, video surveillance of the school’s operations. They found that an index of five characteristics—frequent teacher feedback, data-guided instruction, tutoring, high standards for students, and additional instruction time—explained nearly half the variation in measured charter school quality.[16] These characteristics are often found in “no excuses” charter schools.[17]

Meanwhile, Dobbie and Fryer (2013) find no significant correlation between measured charter school effects and inputs, such as whether the pupil-teacher ratio is less than 13 to 1, whether per-pupil spending is greater than $15,000,[18] or whether more than 11% of teachers have an advanced degree. The one school input found to be significantly related to charter school effectiveness: schools where more than 89% of teachers are uncertified perform significantly worse in math and ELA.

Differences in financial resources could influence the effectiveness of charters, relative to traditional public schools and other charters. Here, too, the empirical research may be dated. In 2011, the tuition formula for state funding to charter schools was frozen, with some allowances for supplemental aid in later years. But in April 2017, New York State increased charter school funding by $500 per pupil, tied future charter school funding to changes in traditional public school spending, and increased the amount of capital funding for charters not located in a traditional public school building.[19] As a result, previous comparisons between the funding of charter and traditional public schools do not necessarily reflect the current state of charter school finance in New York City (Belfield 2017).

Nevertheless, some aspects of the academic literature remain pertinent. In particular, the gains produced by charter schools—relative to the city’s traditional public schools and to one another— would have different implications if they come at the same, lower, or higher spending levels than in the current (post–April 2017) system.

Taxpayers fund New York City charter schools in several ways. The largest source of public funding— intended to cover operating costs—comes from a per-pupil allocation that is set by the state but paid by the school district through a pass-through fund. Charters can receive allocations for other expenses, though these costs tend to be minor. About two-thirds of charter schools also operate rent-free in a traditional public school building,20 while charters that do not operate in a traditional public school building often receive other in-kind support, such as food service and transportation.

New York City charter schools receive, on average, less public support per pupil than do traditional public schools (IBO 2010, IBO 2017, Baker and Ferris 2011, Maloney and Wolf 2017). The difference in total public funding between charter and traditional public schools in New York is substantial, depending on whether the school is located in a traditional public school building. The most recent estimate from the city’s Independent Budget Office (IBO 2017) found that, in 2016–17, the average per-pupil expenditure in New York’s traditional public schools was 5.7% higher than for charters in a traditional public school building, 10.2% higher than for charters that received additional resources for their lease payments, and 24.2% higher than for charters that received no capital spending supplement.

“Fair student funding,” the funding formula for New York City’s traditional public schools, weights resources according to student characteristics, such as whether the student is eligible for free lunch and whether the student is classified as an English language learner (ELL). Baker and Ferris (2011) observe that proportionally fewer charter school students fall into each of these categories than do students in traditional public schools. Thus, if charter schools were funded according to the city’s funding formula for its traditional public schools, they would, on average, be expected to receive fewer dollars per pupil than the average city traditional public school.

Still, the difference in the proportion of charter and traditional public school students classified as ELL has dropped from 14 percentage points at the time of the Baker and Ferris study, to only about 7 percentage points today (New York City Charter School Center 2017). Thus, the cost-adjusted funding difference found by Baker and Ferris is likely—I am not aware of a recent update to this calculation—to be less important today, even as longitudinal results from matching studies suggest that average charter school effects have remained relatively consistent over time.

Charter schools also receive resources from nonpublic sources, especially private foundations. Using data from audited annual financial reports from each charter school, Maloney and Wolf (2017) find that, on average, New York City charter schools received $349 per pupil from nonpublic entities, including $282 per pupil from private philanthropic sources. They also find that New York City’s traditional public schools received $841 per pupil from nonpublic sources. As such, philanthropic funding does not appear to be a primary driver of the generally positive effects of charter schools relative to traditional public schools in New York.

The widespread perception that charter schools receive substantial nonpublic resources to supplement their activities appears to be driven by outliers. Maloney and Wolf find that the vast majority of philanthropic support for charters went to the top 25% of fund-raising schools. Using data from 2008 IRS 990 forms, Baker and Ferris (2011) similarly find that the amount of additional resources from private sources varies dramatically by charter school. For example, Harlem Village received more than $14,000 per pupil, while many other charter schools receive little, if any, philanthropic support.

Further, the empirical evidence offers little guide to whether variation in the amount of nonpublic resources available to a charter school has a meaningful impact on performance. For example, Baker and Ferris find no significant relationship between charter school spending (including public and private sources) and test scores—though this analysis uses aggregate test scores adjusted for student characteristics as a measure of charter school quality, which is not a design that could plausibly lead to a causal estimate of charter school effects.

There is a common claim, supported by numerous anecdotes reported in the media, that a main reason for the higher test scores in charter schools comes from their systematic removal of their most difficult to educate students. It is certainly plausible that there have been cases when charter schools (as well as traditional public schools) have not appropriately handled situations with individual students. However, the academic research literature strongly suggests that charter schools have not systematically manipulated their enrollments in this manner.

Overall, charter school students are as likely, or less likely, to leave their school for another school in the city—or to leave the school district entirely—as are students in traditional public schools. IBO (2015) compares the mobility patterns of students who, in the fall of 2008, enrolled in a charter school or in the nearest traditional public school. It finds that, three years after entering school, about 64% of students who originally enrolled in a charter remained in that school four years later, compared with 56% of students who originally enrolled in a traditional public school. Using student-level data from grades 3–8 from 2007 to 2012, Winters, Clayton, and Carpenter (2017) find that the overall yearly attrition between the sectors was very similar but that attrition from charters was significantly lower than attrition from traditional public schools after accounting for student characteristics.

Winters, Clayton, and Carpenter evaluate whether charter schools systematically remove students who are not performing well on standardized tests. They use student-level data for 2007–12 to compare the attrition patterns of students in New York City charter and traditional public schools overall and by test scores. They find that low-performing students are much more likely to exit their school than are higher-performing students, regardless of the sector. The relationship between students’ test scores and the probability that they left their school did not vary depending on whether students attended a charter or a traditional public school. Because Winters, Clayton, and Carpenter find that attending a charter school reduced the likelihood that a student exited his or her school, they ultimately find that low-performing students in New York City are significantly less likely to exit charter schools than traditional public schools.

Figure 4 illustrates the proportion of students in each sector in a variety of categories, as of 2015–16. A substantially higher proportion of charter school students are black, and substantially fewer are white or Asian relative to the city’s traditional public schools. In addition, the proportion of students with disabilities and those who are ELL is higher in traditional public schools than in charter schools.

UFT (2010) and Baker and Ferris (2011) observe that the proportion of students in charter and traditional public schools who are eligible for free lunch (a measure of extreme poverty)—rather than simply reduced-priced lunch—is substantially larger in traditional public schools than in charter schools. According to the New York City Department of Education (DOE), the proportion of students eligible for free lunch in charter schools is undercounted.22 Instead, Figure 4 uses an “economic need index,” a measure of socioeconomic status that DOE uses to find peer schools with similar student characteristics for its accountability system (a higher value on the index represents a more impoverished student population). According to the index, students who attend charter schools are more disadvantaged, on average, than those who attend traditional public schools.

Figure 5 illustrates some changes in the demographic characteristics of New York City charter schools since 2011–12. In recent years, it appears that the population of students enrolled in charters has become less black, more Hispanic, and with more ELL. Though the demographic enrollment gaps between charters and traditional public schools (Figure 4) are significant, current enrollment trends appear likely to reduce them.

The proportion of students in charter and traditional public schools who have a particular “classification,” such as a disability or ELL status, can be calculated by dividing the number of students with the classification by the total number of students in the sector. Any difference between the two sectors in the proportion of students with a particular characteristic must therefore be driven by a factor that contributes to the numerator and/or denominator of that calculation. Five factors are relevant:

Such students could be less likely to enter a charter school because…

1. They are less likely to apply.


2. After applying, they are less likely to win the lottery (i.e., the lottery is rigged).


3. After winning the lottery…



4. They are less likely to enroll.


5. After entering the school…



6. They are more (or less) likely to experience a change in their classification status.


7. They are more (or less) likely to leave charter schools.

Start with the factors that do not appear to meaningfully affect enrollment gaps. Are lotteries rigged (factor 2, above)? The RFT studies demonstrate that there is no significant relationship between the characteristics of students participating in the lottery and the characteristics of the students who were randomly offered a seat. This suggests that the lottery process is fair and does not influence enrollment gaps between sectors.

Are students with particular characteristics who win the lottery less likely to enroll (factor 3)? Charter schools, for instance, could inappropriately encourage students with particular characteristics to decline their acceptance offers. Of the RFT studies, only Hoxby and Murarka (2009) fully compare the characteristics of students offered a charter seat with those of students who actually enroll. They find no significant difference between accepted applicants and students who enroll— for gender, race/ethnicity, age at time of application, eligibility for free or reduced-price lunch, ELL status, or entering math or reading scores. They do find that students with disabilities are significantly less likely to accept a charter school enrollment offer, though the size of the difference is not substantial (7.61% of lottery winners were in special education, compared with the 7.03% of lottery winners who enrolled).

Unterman (2017) does not present a full set of these descriptive statistics in her RFT analysis of Success Academy schools. She does note that 51% and 43% of lottery winners were Hispanic or black, respectively; and of students who actually enrolled, 45% were Hispanic and 40% were black. Unfortunately, she does not compare the difference in the proportion of lottery winners with ELL status with those who actually enroll.

Are students with particular classifications more likely to leave charter schools (factor 5)? IBO (2014) and IBO (2015) find higher rates of attrition in traditional public schools than in charter schools when students were compared by gender, race/ethnicity, eligibility for free or reduced-priced lunch, ELL status, and disability status.[23] Winters (2013) and Winters (2014a) similarly find that students with disabilities or ELL status were less likely to leave their school if they originally enrolled in a charter than if they originally enrolled in a traditional public school. In other words, different rates of student attrition are an unlikely cause of enrollment differences between the sectors.

Are students with particular classifications in charters more likely to experience a change in their classification status (factor 4)? Winters (2013) finds that, overall, charters and traditional public schools were equally likely to newly classify a student as having a disability. He also finds that charters were significantly more likely to declassify a student out of special education in early elementary grades—which partly drove the widening gap between the two sectors in the proportion of students enrolled in special education in those same grades.

Winters (2014a) finds that the proportion of ELL students in 2010 who no longer had the ELL classification in 2011 was significantly higher in charter schools than in traditional public schools—a result likely influenced by the fact that the average student classified as ELL in a charter was more advanced in English than the average ELL student in a traditional public school. Winters also finds that because the initial proportion of charter school students with ELL status was much smaller than the proportion in traditional public schools, the ultimate impact of declassification was to reduce the special-education gap as students progressed through school.[24] He finds that differences in declassification across sectors were largely responsible for the narrowing of the ELL gap as students progressed through grades.

The research evaluating student enrollment data suggests that differences in the characteristics of charter and traditional public school students are not substantially influenced by factors that occur once students enroll in school. If factors 2–5 do not explain the difference in the characteristics of students in charter and traditional public schools, factor 1—that some types of students are less likely to apply to charters—must explain it. And it does.

All of the RFT studies find that students who applied to New York City charter schools were far more likely to be black—and less likely to be Hispanic or ELL—and have lower standardized test scores at entry than were students entering traditional public schools. There does appear to be considerable variation in special-education applications. Hoxby and Murarka (2009), for instance, find that charter applicants were significantly less likely to be in special education than were other students in the city. Yet Dobbie and Fryer (2013) find no meaningful difference between the special-education status of applicants and other students in the city. Corcoran and Cordes (2015) even find that applicants to Democracy Prep charter schools were more likely to be in special education than was the average student in the city.

Enrollment data on the universe of city schools also suggest that the application stage is the source of the difference in the proportion of students classified as ELL or having a disability. Winters (2013) and IBO (2015) find large differences in the proportion of students in special education across the sectors at the point when students enter kindergarten. Winters (2014a) and IBO (2014) find large differences across the sectors in the proportion of new kindergarten entrants with ELL status. If charter lotteries are fair and certain types of students are not more likely to enroll after winning a lottery seat, the relatively fewer number of disabled and ELL students who enter charter schools is a product of relatively fewer applications by such students.

An expansive charter sector has the potential to affect not only its own students but also those who remain in the traditional public school sector. Some suspect that competition from charter schools incentivizes traditional public schools to improve in order to retain students. Others argue that charter schools harm traditional public schools by siphoning away the most talented students and the education dollars that come with them.

Numerous studies evaluate the effects of expanding charter sectors on student performance in traditional public schools. But only three such studies exist for New York City. These studies use a differences-in-differences design, which evaluates whether there was a change in the trajectory of student performance in a traditional public school that coincides with changes in the competition that it faced from charter schools.

There are various ways to measure such competition. Cordes (2017) uses a geographic measure—specifically, whether any charter school operated within one mile of a traditional public school in a given year. Winters (2012) measures competition as the proportion of a traditional public school’s students who left for a charter school at the end of the previous year. Winters also examines a wider number of grades, while Cordes examines a longer period, though both conclude their evaluations between 2009 and 2011.

Winters finds that increased competition from charter schools had no significant effect on student performance in traditional public schools. The estimated impacts were near zero and were precise. Cordes finds a small, statistically significant, increase in student performance in math and ELA. The positive effect increased as the traditional public school’s distance to the nearest charter school decreased.

In the current environment of far more charter schools, their effect on traditional public schools may be different from that found in these studies. Still, the empirical research has found no evidence of charter schools negatively affecting the quality of traditional public schools— only evidence of a positive effect or no effect.

If charter schools have a positive impact on the performance of traditional public schools, consider some of the implications. Figures 1–2 suggest that the effect of attending a New York City charter school on student performance has remained relatively stable over time. These estimates compare the outcomes of charter school students with those in traditional public schools. Thus, as traditional public schools improve, the effectiveness of charter schools must similarly improve for the impact of attending a charter school instead of a traditional public school to remain constant over time.

In addition to competing for students and education dollars, New York charter schools could affect the performance of the city’s traditional public schools though colocation (sharing the same school building). About two-thirds of the city’s charters colocate with at least one traditional public school.[25] Two studies estimate the impact of colocation with a charter school on the performance of students attending the colocated traditional public school. Both utilize a differences-in-differences design, which tracks student performance in a school before and after a charter is colocated.

Winters (2014) finds that colocation with a charter school had no significant effect on the math or ELA test scores of students attending the colocated traditional public school. Cordes (2017) finds statistically significant test-score gains for students in the colocated traditional public school—indeed, the gains were even larger than those experienced by students attending charters (i.e., the matching-based estimates described above).

Cordes digs further in an effort to pinpoint the factors that caused the gains. She finds that competition from charters leads to a significant reduction in traditional public school enrollment, but such competition does not have a large effect on the characteristics of students. She finds that the loss of students to charters leads to a significant increase in per-pupil expenditures in traditional public schools (for reasons explained above), even as overall spending declines (at least in the short run).

Student performance on standardized tests has been the overwhelming focus of the aforementioned research.[26] Unfortunately, there are no causal estimates of the effect of New York City charters on, say, high school and college graduation rates or on other life outcomes, such as earnings, incarceration, and teen pregnancy.

Enrollment “backfill” (replacing students in charters when they leave for other schools) deserves more research, too. New York City charters are not required to backfill, and many do not: they worry that backfilling would change their school’s carefully cultivated student culture and thus, harm its effectiveness. If this fear is justified, it has important implications for charter school policy and for interpreting charter school effects.

Other areas for future empirical research include measuring the effect of the charter sector on New York City’s teaching workforce, as well as on the (often poor) neighborhoods where charters are most numerous.

The empirical research evaluating New York City’s charter school sector is not perfect. But it is not ambiguous, either: charter schools have had positive effects on the students whom they educate, and they have had a positive or no effect on the quality of the city’s traditional public schools.

Still, further research on the New York City charter school sector is needed. Much of the existing evidence should be updated to ensure that the effects found in earlier studies have remained, even as the sector has expanded rapidly. More research is needed to understand the exact mechanisms that produce these (largely) positive effects, the broader impact that charters have had on students and the city, and whether the lessons that we have learned about the charter sector might improve New York’s traditional public schools.

Effect of Attending a New York City Charter School

Caroline M. Hoxby and Sonali Murarka, “Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement,” NBER Working Paper No. 14852, Apr. 2009. Hereinafter: Hoxby and Murarka (2009)

Sean Reardon, “Review of ‘How New York City’s Charter Schools Affect Achievement,’ ” Education and the Public Interest Center & Education Policy Research Unit, Nov. 12, 2009.

Will Dobbie and Roland G. Fryer Jr., “Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone,” American Economic Journal: Applied Economics 3, no. 3 (July 2011): 158–87. Dobbie and Fryer (2011)

Center for Research on Educational Outcomes at Stanford University, “Charter School Performance in New York City,” Feb. 20, 2013. CREDO (2013)

Will Dobbie and Roland G. Fryer Jr., “Getting Beneath the Veil of Effective Schools: Evidence from New York City,” American Economic Journal: Applied Economics 5, no. 4 (Oct. 2013): 28–60. Dobbie and Fryer (2013)

Sean P. Corcoran and Sarah A. Cordes, “The Continuing Impact of Democracy Prep Public Schools: Preliminary Report,” New York University, July 2015. Corcoran and Cordes (2015)

Rebecca Unterman, “An Early Look at the Effects of Success Academy Charter Schools,” MDRC, Aug. 2017. Unterman (2017)

Center for Research on Educational Outcomes at Stanford University, “Charter School Performance in New York City,” Oct. 2017. CREDO (2017)

Effect of New York City Charter Schools on Student Performance in Traditional Public Schools

Marcus A. Winters, “Measuring the Effect of Charter Schools on Public School Student Achievement in an Urban Environment: Evidence from New York City,” Economics of Education Review 31, no. 2 (Apr. 2012): 293–301. Winters (2012)

Marcus A. Winters, “The Effect of Co-Locations on Student Achievement in NYC Public Schools,” Manhattan Institute, Feb. 25, 2014. Winters (2014

)

Sarah A. Cordes, “In Pursuit of the Common Good: The Spillover Effects of Charter Schools on Public School Students in New York City,” Education Finance and Policy, July 2017. Cordes (2017)

Issues Related to Charter School Enrollments

Hoxby and Murarka (2009).

Sean Reardon, “Review of ‘How New York City’s Charter Schools Affect Achievement,’ ” Education and the Public Interest Center & Education Policy Research Unit, Nov. 12, 2009.

United Federation of Teachers, “Separate and Unequal: The Failure of New York City Charter Schools to Serve the City’s Neediest Students,” Jan. 2010. UFT (2010)

Independent Budget Office of the City of New York, “Comparing the Level of Public Support: Charter Schools Versus Traditional Public Schools, Feb. 2010. IBO (2010)

Dobbie and Fryer (2011).

Bruce D. Baker and Richard Ferris, Adding Up the Spending: Fiscal Disparities and Philanthropy Among New York City Charter Schools (Boulder, CO: National Education Policy Center, 2011). Baker and Ferris (2011)

Jack Buckley and Carolyn Sattin-Bajaj, “Are ELL Students Underrepresented in Charter Schools? Demographic Trends in New York City, 2006–2008,” Journal of School Choice 5 (2011): 40–65.

Winters (2012).

Robin Lake, Betheny Gross, and Patrick Denice, “New York State Special Education Enrollment Analysis,” Center for Reinventing Public Education, Nov. 2012.

Center for Research on Educational Outcomes at Stanford University, “Charter School Performance in New York City,” Feb. 20, 2013. CREDO (2013)

Marcus A.Winters, “Why the Gap? Special Education and New York City Charter Schools,” Center for Reinventing Public Education & Manhattan Institute, Sept. 30, 2013. Winters (2013)

Dobbie and Fryer (2013).

Stephen Eide, “Should Charter Schools Pay Rent? Implications for Staffing and Growth,” Manhattan Institute, Feb. 6, 2014. Eide (2014)

Winters (2014).

John Kucsera with Gary Orfield, “New York State’s Extreme School Segregation,” Civil Rights Project, Mar. 2014.

Marcus A. Winters, “Why the Gap? English Language Learners and New York City Charter Schools,” Manhattan Institute, Oct. 6, 2014. Winters (2014a)

New York City Charter School Center, “Still Building Inequality: How New York’s Charter Public School Students Continue to Bear the Cost of School Facilities,” Nov. 2014.

Independent Budget Office of the City of New York, “Comparing Student Attrition Rates at Charter Schools and Nearby Traditional Public Schools,” Jan. 2015. IBO (2015)

Susan Aud Pendergrass and Nora Kern, “Waiting for Their Chance: A Closer Look at Wait Lists in Urban Public Charter Schools,” National Alliance for Public Charter Schools, May 4, 2015

Corcoran and Cordes (2015).

Center for Research on Educational Outcomes at Stanford University, “Urban Charter School Study: Report on 41 Regions,” 2015.

Democracy Builders, “No Seat Left Behind: The Unfilled Potential of Empty Seats in Charter Schools,” 2015.

New York City Charter School Center, “New York City Charter Schools: 2016–17 Enrollment Lottery Estimates,” June 2016.

Halley Potter, “Diversity in New York City’s Universal Pre-K Classrooms,” Century Foundation, Sept. 20, 2016.

New York City Charter School Center, “Neighborhoods Primed for Charter School Growth,” 2016.

Independent Budget Office of the City of New York, “Staff Costs and Pass-Through Payments to Charter and Nonpublic Schools Propel Education Department Spending,” Feb. 2017.

Marcus A. Winters, Grant Clayton, and Dick M. Carpenter II, “Are Low-Performing Students More Likely to Exit Charter Schools? Evidence from New York City and Denver, Colorado,” Economics of Education Review 56 (Feb. 2017): 110–17.

Independent Budget Office of the City of New York, “With State Formula for Charter School Funding Likely to Change, City Costs to Grow More than Budgeted,” Mar. 2017. IBO (2017)

Cordes (2017).

Larry D. Maloney and Patrick J. Wolf, “Charter School Funding: Inequity in New York City,” University of Arkansas, Aug. 2017. Maloney and Wolf (2017) Unterman (2017).

CREDO (2017).

Clive Belfield, NEPC Review: Charter School Funding: Inequity in New York City (Boulder, CO: National Education Policy Center, 2017). Belfield (2017)

New York City Charter School Center, “English Language Learners and NYC Charter Schools,” 2017.

New York City Charter School Center, “New York City Charter Schools Are in Demand...and Growing,” 2017.

New York City Charter School Center, “Students with Disabilities and NYC Charter Schools,” 2017.

 

1. I have, to my knowledge, included all relevant direct analyses of New York City’s charter school sector in this review. If I have missed any relevant research, that is my regrettable error. I can say with confidence, however, that whatever research I may have missed has not played a meaningful role in New York’s charter school conversation—it is neither cited in the academic literature nor has it influenced the policy conversation.
2. For instance, a 2003 report by the Institute for Education and Social Policy at New York University considered whether a charter school authorizer should use its authority to close ineffective charter schools. The issue of accountability for the charter sector remains important. However, the context for that conversation has changed dramatically since 2003. Thus, I do not include that paper in this review.
3. See, e.g., the United Federation of Teachers, New York City Charter School Center, Independent Budget Office, and the National Education Policy Center.
4. The seven studies are: Caroline M. Hoxby and Sonali Murarka, “Charter Schools in New York City: Who Enrolls and How They Affect Their Students’ Achievement,” NBER Working Paper No. 14852, Apr. 2009; Will Dobbie and Roland G. Fryer Jr., “Are High-Quality Schools Enough to Increase Achievement Among the Poor? Evidence from the Harlem Children’s Zone,” American Economic Journal: Applied Economics 3, no. 3 (July 2011): 158–87; Center for Research on Educational Outcomes at Stanford University, “Charter School Performance in New York City,” Feb. 20, 2013; Will Dobbie and Roland G. Fryer Jr., “Getting Beneath the Veil of Effective Schools: Evidence from New York City,” American Economic Journal: Applied Economics 5, no. 4 (Oct. 2013): 28–60; Sean P. Corcoran and Sarah A. Cordes, “The Continuing Impact of Democracy Prep Public Schools: Preliminary Report,” New York University, July 2015; Rebecca Unterman, “An Early Look at the Effects of Success Academy Charter Schools,” MDRC, Aug. 2017; and Center for Research on Educational Outcomes at Stanford University, “Charter School Performance in New York City,” Oct. 2017.
5. Using simulations and data for Boston, Walters finds evidence of an inverse relationship between preference for charter schooling and the effect of attending a charter school—i.e., within the Boston context, it appears that the students least likely to apply to charter schools would benefit from them the most. See Christopher R. Walters, “The Demand for Effective Charter Schools,” NBER Working Paper No. 20640, Feb. 2017.
6. Each of these studies reports several different estimates. I focus on treatment-on-the-treated estimates that evaluate the effect of attending a charter. When a paper examines multiple grades, I report a representative finding. I also limit my discussion to the overall effects of charter schools and not how effects vary by student or school characteristics.
7. An estimate is said to be statistically significant if we are at least 95% confident that the effect, whether positive or negative, is not zero. Statistical significance takes into account both the magnitude of an estimate and the precision with which it is estimated. An estimate of small magnitude might be statistically significant if it is estimated precisely, while an estimate of large magnitude might be statistically insignificant if it is estimated imprecisely. The precision of the estimate is affected by the variability of the result, as well as by the number of observations included in the analysis.
8. Each study reports a variety of estimates, to confirm the results’ robustness or to find subtle differences in effect. I focus on the estimate that best reflects the overall finding of the respective study. However, my choice of the primary estimate is not always the same as the author’s choice. For example, the CREDO reports highlight estimates that combine four years of observations together; yet for the purposes of this review, the most relevant estimates are for each year separately.
9. Six of the seven studies report the effect on a per-year basis. Unterman (2017) reports the effect after a period of four years. I translated the latter result into a per-year estimate by dividing it by four.
10. The Krueger estimates suggest that the effect of small class sizes was mostly attributable to the first year of the treatment. See Alan B. Krueger, “Experimental Estimates of Education Production Functions,” Quarterly Journal of Economics 115, no. 2 (May 1999): 497–532. CREDO (2013) and CREDO (2017) translate effects from standard deviation units into “days of additional learning.” According to this translation, an increase of 0.0017 standard deviations represents the equivalent of an additional day of schooling. To find the estimated increase in days of schooling, one would multiply the estimate in standard deviation units by 0.0017. This type of translation has gained some popularity: it is somewhat reasonable, and it is easily understandable for the lay reader. I encourage the reader to perform this calculation if it helps to put the overall effect into context. Nonetheless, the “days of additional learning” translation is controversial and, at best, imperfect, so I have not presented the results in this way.
11. See n. 7 above.
12. See n. 8 above.
13. See n. 9 above.
14. See n. 10 above.
15. Likewise, estimates from Cohodes, Setren, and Walters find that the average effect of Boston’s charter school sector has not diminished—and has likely increased—even as it doubled its market share in that city. See Sarah Cohodes, Elizabeth Setren, and Christopher R. Walters, “Can Successful Schools Replicate? Scaling Up Boston’s Charter School Sector,” School Effectiveness & Inequality Initiative, Oct. 2016.
16. Dobbie and Fryer (2011) find that when these activities were adopted in Houston’s traditional public school system, they led to similarly positive effects in student achievement.
17. Angrist, Pathak, and Walters (2011) find that “no excuses” schools with similar characteristics are much more effective than other schools in Boston.
18. It is not clear from the paper whether the per-pupil spending measure includes expenditures from public and private sources.
19. Monica Disare, “After Heated Debate, New York Charter Schools Receive Boost; School Aid Increases by $1.1 Billion,” Chalkbeat, Apr. 10, 2017.
20. New York City Charter School Center, “Facilities.”
21. See New York City Department of Education, “Data About Schools.” For the economic need index, see New York City Department of Education, “2016–17 School Quality Reports.” The index is calculated as: (% temporary housing) + (% HRA-eligible x 0.5) + (% free-lunch-eligible x 0.5). HRA-eligible refers to students whose families receive certain types of public assistance, as identified by the New York City Human Resources Administration.
22. “Approximately 25% of charter schools in NYC do not utilize NYC DOE School Food to provide meal services. The NYC DOE Office of School Food does not collect documentation on students’ eligibility for free or reduced price lunch from schools that do not utilize NYC DOE school food. As a result, the poverty figures may be understated for approximately 25% of charter schools.” See New York City Department of Education, “Demographic Snapshot.”
23. IBO (2014) reports that students with disabilities were much more likely to exit charter schools than traditional public schools. This finding was the result of a data error and was corrected in IBO (2015). See Sarah Darville, “IBO Admits Charter School Special Ed Attrition Numbers Missed the Full Picture,” Chalkbeat, Feb. 19, 2014.
24. As far as the gap narrowing, Winters (2014a) observes: “Consider a simple, stylized example, one with 100 students in the charter sector and another with 100 students in traditional public schools. In charters, two of the 100 students (2%) are classified as ELL, compared with 20 of 100 students (20%) in traditional public schools. If, hypothetically, the next year charter schools declassified one of their ELL students, charters would have declassified 50% of their ELL population that year; if traditional public schools declassified five of their ELL students, traditional public schools would have declassified 25% of their ELL population, respectively. Charters would therefore have a larger rate of declassification. Yet in this example, the ELL gap has nonetheless closed from 18% (20%–2%) to 14% (15%–1%). The same holds true for the actual data discussed earlier: the impact of charter schools declassifying a larger share of their ELL students each year is to reduce the ELL gap over time.”
25. New York City Charter School Center, “Facilities.”
26. Corcoran and Cordes (2015) measure the effect of attending a Democracy Prep school on student attendance, and Dobbie and Fryer (2013) examine attendance and on-time matriculation. These metrics are worth measuring but cannot be considered educational outcomes.