Sarah Komisarow - Recherche en IA spécialisée au MIT

Sarah Komisarow

Department of Public Policy

and Economics

Sanford School of Public Policy

Duke University

Durham, Caroline du Nord 27708

sarah.komisarow@duke.edu

COMPREHENSIVE SUPPORT AND STUDENT

SUCCESS: CAN OUT OF SCHOOL TIME

MAKE A DIFFERENCE?

Abstrait
StudentU is a comprehensive program that provides education,
nutrition, and social support services to disadvantaged middle
and high school students outside of the regular school day. Dans ce
paper I investigate the eﬀects of this multiyear program on the
early high school outcomes of participating students by exploiting
data from oversubscribed admissions lotteries. I ﬁnd that the sub-
group of lottery winners who entered the comprehensive program
with low baseline achievement earned more course credits (0.82
credits), achieved higher grade point averages (0.37 grade points),
and were less likely to be suspended (17.1 percentage points) dur-
ing ninth grade than their lottery loser counterparts. Enquête
of intervening variables indicates that on-time grade progress and
decreases in course failure and disciplinary infractions are po-
tential mediating channels. Using an index of early high school
résultats, I predict that lottery winners are around 4 percentage
points more likely to graduate from high school than lottery losers
(5 percent eﬀect). These results suggest that comprehensive ser-
vices delivered outside of the regular school day have the potential
to improve the educational outcomes of disadvantaged students.

https://doi.org/10.1162/edfp_a_00366

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

579

Comprehensive Support and Student Success

INTRODUCTION

1 .
Gaps in skills between children born into low- versus high-income families exist at all
levels of K–12 schooling in the United States. In some cases, these gaps appear prior
to kindergarten entry and then widen as students progress through the K–12 system
(Duncan and Magnuson 2011; Farkas 2011; Reardon 2011). Long-studied policy interven-
tions designed to ameliorate income-based skill gaps frequently take aim at the existing
K–12 system. These include school resource policies, input policies (par exemple., class size, dans-
novative curricula, and high-quality teachers), and other market-based reforms (par exemple.,
charter schools, voucher programs). In contrast, much less attention has been devoted
to understanding the eﬀects of interventions that target another point of intervention:
time spent outside of school. Structured activities and services provided outside of the
regular school day—typically after school or during the summer—are increasingly the
focus of public investment in the United States, but little is known about their eﬀects
on student outcomes. Chiffre 1 depicts appropriations between 1995 et 2018 for the
main federal program that serves children outside of the regular school day in the
États-Unis, the 21st Century Community Learning Center program.1 Despite a nearly
one thousand–fold increase in real federal support for this program over the past two
decades, very little is known about how time spent outside of school aﬀects student
success.

Existing program evaluation research literature that addresses time spent and ser-
vices provided outside of school typically focuses on specialized interventions that
narrowly target speciﬁc skill domains or areas of student need among disadvantaged
students.2 Examples of specialized interventions that target speciﬁc skills include aca-
demic tutoring (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina
2010; Heinrich et al. 2014), advising, coaching, and mentoring (Grossman and Tierney
1998; Angrist, Lang, et Oréopoulos 2009; Bettinger and Baker 2014; Castleman, Page,
and Schooley 2014), summer reading programs (Kim and Quinn 2013), and behavioral
supports (Heller et al. 2013; Cook et al. 2014; Heller et al. 2017). Related interventions
address speciﬁc areas of student need, such as nutrition, although these interventions
are typically carried out within the regular school day (Bhattacharya, Currie, and Haider
2006; Imberman and Kugler 2014; Corcoran, Elbel, and Schwartz 2016).

In contrast, much less is known about the eﬀects of intensive interventions that
take a broader view and provide comprehensive education, nutrition, and social sup-
port services in a uniﬁed setting outside of the regular school day and outside of the
existing K–12 system.3 Two existing papers in the literature attempt to address this

1. The program studied in this paper is a 21st Century Community Learning Center. The 21st Century Community
Learning Center grant program is the largest competitive federal grant program devoted to funding out-of-
school time programs and activities for children in the United States (Augustine and Thompson 2020).
2. McCombs et al. (2019) provide a review of studies that focus on summer intervention and note that the majority
of existing work in this area addresses the eﬀectiveness of summer interventions that focus on reading and
literacy.

3. Three related but distinct strands of literature consider (1) the eﬀects of programs and services provided to
children and families through coordinated, community eﬀorts but still accessed individually based on individ-
ual families’/children’s needs (voir, par exemple., Dobbie and Fryer 2011); (2) the eﬀects of “wraparound” or supportive
services targeting students’ nonacademic needs that are delivered through targeted schools or districts and in-
tegrated with existing school/district services (voir, par exemple., Johnston et al. 2017; Winters 2017; Gandhi et al. 2018;
Opper et al. 2019); et (3) the eﬀects of voluntary, district-oﬀered summer programming (Augustine et al.
2016).

580

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Sarah Komisarow

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

Remarques: This figure depicts federal appropriations (dans 2017 dollars) for the Twenty-First Century Community Learning Center Program,
which was authorized by Title I, Part X of the Elementary and Secondary Education Act (ESEA). This program provides funding for
schools in the United States to implement activities that provide educational, recreational, cultural, health, and social services outside
of the regular school day. For more information, see Stapleton (1998) and McCallion (2001).

Chiffre 1. Federal Appropriations for 21st Century Community Learnings Centers in the United States, 1995—2018

question, although they do so imperfectly because the programs tested paired the pro-
vision of intensive, comprehensive services with generous ﬁnancial incentives, lequel
have been shown to have strong, independent eﬀects on student outcomes, tel que
achievement and high school graduation, in some other contexts (Leuven, Oosterbeek,
and Klaauw 2010; Ford and Kwakye 2016). Oreopoulos, Brun, and Lavecchia (2017) concernant-
port quasi-experimental evidence on the eﬀects of the Pathways to Education program,
an intensive and comprehensive, out-of-school program for public housing residents
in Toronto, Canada. They report that access to Pathways for the duration of high school
increased graduation rates by between 5 et 15 percentage points and postsecondary
enrollment by between 4 et 19 percentage points. In related work, Rodriguez-Planas
(2012) reports results from a multi-site randomized controlled trial of the Quantum
Opportunity Program (QOP), which provided intensive and comprehensive services
outside of the regular school day to low-achieving high school students. She also ﬁnds
that the oﬀer of comprehensive services increased high school graduation rates, à
least in the short run. Cependant, she ﬁnds some evidence to suggest that the control
group eventually caught up and graduated high school at rates similar to their treated
homologues.

In this paper, I address this gap in the literature by producing causal evidence on the
eﬀects of intensive, comprehensive education, nutrition, and support services provided
outside of the regular school day in a uniﬁed setting on the early high school outcomes

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

581

Comprehensive Support and Student Success

of disadvantaged students. I study this question in the context of StudentU, a nonproﬁt
organization in Durham, North Carolina, that provides these services through a multi-
year program that takes advantage of students’ time outside of school after school dur-
ing the academic year and during summer breaks. To produce evidence on the causal
eﬀects of this program, I leverage data from oversubscribed admissions lotteries car-
ried out to allocate slots in the program. By comparing lottery winners and losers, je
produce internally valid estimates of the eﬀect of access to comprehensive services on
the outcomes of disadvantaged students. I take advantage of the multiyear structure of
the program to estimate impacts on student outcomes in the medium term. Speciﬁ-
cally, I examine impacts on credit accumulation, grade point average, and suspension
as broad measures of how the program aﬀects students’ cognitive and noncognitive
skills in early high school.

En moyenne, StudentU lottery winners accumulated more credits and were less likely
to be suspended in early high school than their lottery loser counterparts. Cependant, je
ﬁnd strong evidence to suggest that these mean impacts were exclusively driven by lot-
tery winners who entered the program with low levels of baseline achievement. My in-
vestigation of treatment eﬀect heterogeneity along this dimension suggests that eﬀects
in the full sample of lottery winners were driven by this subgroup. I further ﬁnd that
this subgroup achieved higher grade point averages by the end of ninth grade. Specif-
ically, lottery winners with low baseline achievement accumulated 0.82 more course
credits, achieved higher grade point averages (GPAs) par 0.37 grade points, and were
17.1 percentage points less likely to be suspended during ninth grade.

By the end of ninth grade, StudentU lottery winners who entered the program with
low baseline achievement had credit accumulation and suspension outcomes that were
not only larger on average than their control group counterparts—who had similar
achievement test scores at baseline—but were also larger on average than members
of the control group who entered the program with high baseline achievement. Pour
GPAs, the increase among StudentU lottery winners with low baseline achievement
was large enough to close around 35 percent of the gap between the low- and high-
achieving students in the control group.4 These ﬁndings suggest that comprehensive
services provided outside of the regular school may be a particularly eﬀective strategy
for improving outcomes of the most disadvantaged students.

To provide evidence on the mechanisms underlying these eﬀects I leverage data
on intervening variables to help to distinguish candidate channels of on-time grade
progress, course-taking and completion, student achievement, and behavior in school.
I ﬁnd evidence most consistent with on-time grade progress, course completion, et
improved behavior in school as primary channels. I do not ﬁnd any statistically signif-
icant evidence of increases in test scores, but I acknowledge that low statistical power
is a concern for test scores outcomes. Coupled with the ﬁndings on for whom compre-
hensive services work, these results about mechanisms provide policy-relevant insight
into how comprehensive services work to improve student outcomes.

4. Students in the high baseline achievement group entered the program with an average math and reading score
de 0.38 SD (median = 0.26 SD) relative to the statewide mean of zero (and unit standard deviation). In contrast,
students in the low baseline achievement group entered the program with an average math and reading score
of −0.90 SD (median = −0.80 SD).

582

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Sarah Komisarow

I conclude by presenting the results from back-of-the-envelope calculations de-
signed to investigate how the eﬀects of StudentU on early high school outcomes likely
translate into longer-term eﬀects on in high school graduation. Using an index of early
high school outcomes and data on past cohorts of ﬁrst-time ninth-grade students, je pré-
dict that lottery winners are around 4 percentage points more likely to graduate from
high school than lottery losers. These results are on the low end of the range of eﬀects
reported in the previous literature but are still consistent with previous ﬁndings about
comprehensive programs delivered outside of school. I conclude with a brief discussion
of program costs.

This paper contributes to the growing research literature on the eﬀects of compre-
hensive services provided outside of the regular school day on student outcomes and
builds on this previous work by providing evidence from a setting uncontaminated by
the addition of ﬁnancial incentives for participation. The results from this paper pro-
vide new, policy-relevant insight into the role that nonproﬁt and other organizations
outside of the K–12 system can play in shaping the outcomes of disadvantaged students
outside of the regular school day.

2 . B AC K G RO U N D
StudentU’s Mission
StudentU is a 501(c)(3) nonproﬁt organization that provides education, nutrition, et
related social support services outside of the regular school day to disadvantaged stu-
dents in Durham, North Carolina. These services include academic programming;
healthy meals and snacks; parent/caregiver outreach; coaching, advising, and mentor-
ing; and referrals for children and families to other community services. These compre-
hensive services are delivered through intensive after-school programming that takes
place outside of the regular school day during the academic year and summer program-
ming that occurs between academic years. StudentU’s core values are: Energize Your
Community, Achieve Greatness, Respect Yourself and Others, Discover Your Best Self,
Dream Fearlessly, and Share Your Brilliance. These core values inﬂuence the activities
of the organization and motivate the organization’s mission, which is, “[T]o empower
students in the Durham Public Schools to own their education by developing the aca-
demic skills and personal well-being necessary to succeed in college and beyond.”5

StudentU students are drawn from traditional public schools in Durham Public
Schools (DPS), charter schools, and independent (private) schools located within the
Durham County, North Carolina. Students enter the program as rising sixth graders—
the summer before middle school begins—and remain in the program until high
school graduation. Students in the program participate after school (called “School Year
Program”) and during the summer (called “Summer Academy”). The School Year Pro-
gram takes place at a central location on regular attendance days during the academic
année. Summer Academy occurs for six weeks during the summer.

StudentU selected its ﬁrst cohort of students during the spring semester of the
2006–07 academic year and has selected a new cohort of students in the spring of
every academic year since then. Following information sessions in schools, directed

5. Despite the reference to DPS in the mission statement, StudentU is also open to students who attend charter

schools and independent (private) schools in Durham County.

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

583

Comprehensive Support and Student Success

recruitment through school staﬀ (par exemple., principals, teachers, and counselors), and out-
reach activities conducted by StudentU staﬀ in the community, students are eligible
to apply for admission to StudentU in the fall of ﬁfth grade and then receive notice of
their admissions decision in the spring of ﬁfth grade. Students formally matriculate
into the program during the summer prior to middle school entry (sixth grade). Begin-
ning in the 2011–12 school year, StudentU received applications in excess of its capacity
and used lotteries to determine admission to the program. These annual lotteries were
conducted separately by gender due to the higher proportion of applications received
from female students.6 Upon acceptance into the program, StudentU students and
their families sign contracts to participate in program activities over the subsequent
seven-year period, although in practice sometimes students do not “take-up” the lottery
oﬀer or may leave the program before the end of high school. There are no direct costs
for students or their families to participate, and transportation is provided for students
to attend both the School Year Program and Summer Academy.

Due to the intensity and scope of the comprehensive services provided, StudentU
is intentionally small—there are 50 students in each cohort. All students who submit a
complete application, which includes short essay questions and a teacher recommen-
dation, are screened against two criteria prior to the lottery: students must be the ﬁrst
person in their immediate family to attend college and/or eligible for free/reduced price
lunch. All complete applications that satisfy at least one of these two criteria are entered
into that year’s admissions lottery. In practice, there are often fewer than 50 slots avail-
able each year, since StudentU has historically granted automatic admission to siblings
who have an older sibling already enrolled in the program.

StudentU Programming
Following acceptance into StudentU, rising sixth-grade students attend Summer
Academy, an intensive six-week program designed to strengthen students’ academic
credentials through small-group instruction and to promote students’ development
through elective courses and activities designed to enhance socioemotional learning.
During Summer Academy, which meets from 8 a.m. à 4 p.m. ﬁve days per week for
six weeks, students attend daily, 50-minute classes in Math, Science, English Language
Arts, and Global Connect (Social Studies). These small-group classes are led by trained
local college students (typically those pursuing a baccalaureate degree in education)
or certiﬁed teachers from surrounding local school districts who work for StudentU
during the summer. In addition to these required courses, students also develop their
personal interests by choosing elective courses (par exemple., poetry, the arts, or astronomy). Fi-
enfin, outside of these core and elective courses, StudentU also promotes physical and
emotional health through participation in exercise-based activities and wellness classes.
Each day also includes one hour of “Family Time,” during which students meet with a
regular group of peers and an adult mentor (typically a teacher or other staﬀ member
in the program). Family Time is split into three short sessions dispersed throughout
the day, thus providing students with regular and sustained adult and peer support
throughout the program.

6. See column 7 of table 1.

584

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Sarah Komisarow

Following Summer Academy, StudentU students participate in 15 hours per week of
after school programming each week for 30 weeks during the school year. After school
programming includes tutoring and supervised homework time (“Study Skills”), alors que
also allowing students to participate in extracurricular activities ranging from athletics
to the arts. Students also participate in structured academic activities outside of their
school work each day (“Academic Power Hour”), which are led by program staﬀ and de-
signed to strengthen students’ reading, math, and writing skills. To facilitate the provi-
sion of these services, StudentU has 20 full-time employees, including a social worker,
a bilingual parent-liaison, and an education specialist. In addition to these full-time
staﬀ, additional tutors, mentors, and summer teachers are hired from surrounding
universities and local school districts.

Aside from School Year and Summer Academy programming, StudentU also oﬀers
supportive services to enrolled students and their families. These services—provided by
program staﬀ—include access to counseling, a student support fund (for emergencies),
assistance with accessing community resources, one-on-one or small group instruction
on academic/study skills, assistance for students/families to access school- or district-
provided academic resources, and assistance for families accessing school- or district-
provided support services.

Previous Work
Comprehensive programs designed to address multiple areas of skill development or
student need may be particularly eﬀective at improving educational outcomes among
disadvantaged students because they provide students with access to a suite of services
that have been shown to be eﬀective at improving student outcomes in other settings.
These include, Par exemple, academic support and nutrition assistance (c'est à dire., access to
healthy meals and snacks), which I discuss in turn below. Above and beyond these ser-
vices, which target skill development and speciﬁc areas of need, comprehensive pro-
grams also provide support to students in ways that resemble interventions designed
to provide coaching, advising, and mentoring. This aspect of comprehensive programs
may better allow students to take full advantage of the other services in the program
by providing positive relationships, encouragement, and additional assistance when
needed. In their analysis of the successes of the Pathways program for public housing
residents in Toronto, Oreopoulos, Brun, and Lavecchia (2017) cite qualitative evidence
from staﬀ interviews and quantitative evidence from mediation analysis to emphasize
the importance of these supportive relationships in facilitating the successes of com-
prehensive programs.

Much of the previous work on the eﬀects of academic support and instruction pro-
vided outside of the regular school day ﬁnds that traditional tutoring programs—when
delivered in isolation from other supportive services and in “low doses”—are largely in-
eﬀective at raising student achievement.7 Fryer (2017) reports meta-analytic estimates
of the eﬀects of low-dosage tutoring that are 0.074 standard deviation (standard error =
0.045) in math and 0.050 standard deviation (standard error = 0.045) in reading.

7. Fryer (2017) deﬁnes “high-dosage” tutoring as tutoring delivered in groups containing 6 students (or fewer) pour
3 or more days each week, or for a total of 50 or more hours in a 36-week period. Tutoring programs that meet
for fewer hours or in larger groups are deﬁned as “low-dosage.”

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

585

Comprehensive Support and Student Success

Using quasi-experimental variation induced by the introduction of supplemental ed-
ucation services (SES) (c'est à dire., tutoring) in school districts under No Child Left Behind,
several recent papers investigate the eﬀects of tutoring on student achievement. Hein-
rich, Meyer, and Whitten (2010) report results from the introduction of SES in Mil-
waukee Public Schools. They do not ﬁnd any statistically signiﬁcant eﬀects on student
math or reading gains. In related work, Heinrich et al. (2014) report the results of a
multi-site investigation of the same type of services provided to low-income students
in four large school districts in the United States. Although they ﬁnd some evidence of
positive math and reading gains in one district, they do not ﬁnd consistent evidence of
positive eﬀects on achievement in any of the other three.

In contrast, tutoring programs that deliver intensive instruction in small group set-
tings (sometimes referred to as “high-dosage” tutoring or tutorials in existing literature)
have demonstrated more promising results. Factors such as tutor quality, intensity (jours
and hours per week), group size, program management, and diﬀerentiated instruction
are important determinants of whether tutoring programs are successful. In a study
of SES in Pittsburgh Public Schools, Zimmer, Hamilton, and Christina (2010) report
some evidence of positive test score gains in math and identify diﬀerentiated instruc-
tion by student skill level, emphasis on knowledge gaps from previous coursework, et
tutor experience as key factors aﬀecting success. Fryer (2017) reports meta-analytic es-
timates of the eﬀects of high-dosage tutoring that are 0.393 (standard error = 0.095) dans
math and 0.405 (standard error = 0.047) in reading. In related work, Ander, Guryan,
and Ludwig (2016) report the results from several randomized controlled trials of in-
tensive “tutorials” (two-on-one tutoring carried out within the regular school day) sur
the achievement of low-income students. These tutorials, which provide individual-
ized, daily instruction to struggling students, have been shown to increase math test
scores by 0.23 standard deviation and reduce the incidence of math course failure by
as much as 50 pour cent. Conﬁrmatory quasi-experimental evidence demonstrates that
high-dosage tutoring inside and outside of the regular school day is particularly eﬀec-
tive at raising student achievement in math (Fryer 2014; Kraft 2015; Chabrier, Cohodes,
et Oréopoulos 2016).

The evidence on the eﬀects of nutrition assistance on student outcomes is more
mixed, with some papers reporting positive eﬀects and others reporting null results.
In early evaluation of the School Breakfast Program, Bhattacharya, Currie, and Haider
(2006) report improvements in nutrition outcomes among students exposed to the
program. These included improvements in outcomes like vitamin and mineral intake
and decreases in the poor diet outcomes (par exemple., the percent of calories obtained from
fat). In work investigating the eﬀects of Breakfast in the Classroom (BIC), Imberman
and Kugler (2014) report evidence of positive eﬀects on student test scores. By exploit-
ing variation in the timing of BIC introduction, the authors conclude that the likely
source of these test score gains is improvements in test performance and not necessar-
ily changes in student learning. In related work, Corcoran, Elbel, and Schwartz (2016)
examine the introduction of BIC across schools in New York City, although in that study
the authors ﬁnd no evidence of improvements in student test scores.

Existing literature on the eﬀects of intensive coaching, advising, and mentoring
largely comes from interventions designed to improve educational outcomes for stu-
dents transitioning from high school to postsecondary or from programs for students

586

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Sarah Komisarow

in the early years of their postsecondary careers. Although the contexts of these inter-
ventions are varied, papers in this literature provide insight into the mechanisms that
underlie the successes of these interventions. In a study investigating the eﬀects of
intensive coaching and mentoring on students’ college application behavior and sub-
sequent enrollment, Carrell and Sacerdote (2017) examine how coaching and mentor-
ing can improve student outcomes when students face intricate and multifaceted chal-
lenges that extend over long time horizons (par exemple., the college application process). Ils
argue that intensive coaching and mentoring is likely a substitute for the sustained ef-
forts of a parent, teacher, or other support person. Evidence from Bettinger and Baker
(2014) provides further insight into the ways in which intensive coaching and men-
toring can help students connect their habits, activités, and behaviors to their goals.
The authors hypothesize that coaches, advisors, and mentors help students concretely
develop strategies and implement action plans in service of their abstract goals. In a
randomized controlled trial of coaching for nontraditional college students, the authors
ﬁnd that students exposed to the oﬀer of regular coaching had improved persistence
outcomes in a variety of postsecondary settings. Coaching and mentoring eﬀects on stu-
dent outcomes appear even when the services are provided in small doses. Castleman,
Page, and Schooley (2014) ﬁnd that access to a single 2- to 3-hour session for students
transitioning from high school to college improves student enrollment outcomes. Dans
related work, Clotfelter, Hemelt, and Ladd (2018) ﬁnd that in addition to generous ﬁ-
nancial aid, nonﬁnancial supports (par exemple., mentoring and social support) improved on-
time credit accumulation and grade point averages among low-income, ﬁrst-generation
college students at a large, selective public university.

3 . DATA
The data in this paper come from two sources: (1) StudentU application and lottery
records and (2) administrative education data from the North Carolina Education Re-
search Data Center (NCERDC) covering the universe of students enrolled in public and
charter schools in North Carolina.

Matching StudentU Applications to NCERDC Data
All StudentU application and lottery records were matched to existing NCERDC admin-
istrative data based on the student’s ﬁrst name, last name, date of birth, and elementary
school attended in ﬁfth grade.8 The matching procedure carried out by NCERDC staﬀ
yielded an overall match rate of 86 percent and was similar to the match rate obtained
in other studies that have utilized the same NCERDC data.9 StudentU started using ad-
missions lotteries in the spring of 2012, resulting in three separate lottery-based cohorts
from the years 2012, 2013, et 2014, respectivement. Online appendix Table A1 (available
in a separate online appendix that can be accessed on Education Finance and Policy’s
Web site at https://direct.mit.edu/edfp) summarizes the results from the NCERDC

8. Per institutional review board mandate, StudentU application and lottery records were sent directly from Stu-
dentU to staﬀ at NCERDC, who handled the process of matching the identiﬁable student-level information
contained in lottery records to the administrative student-level data. The matched data were then de-identiﬁed
and transferred from NCERDC to the author for analysis.

9. See Gassman-Pines and Bellows (2018) for a brief overview and discussion of match rates obtained in previous

studies that have utilized the NCERDC data.

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

587

Comprehensive Support and Student Success

matching procedure. After excluding younger siblings who were granted automatic
admission to StudentU outside of the lottery process, 324 out of 376 student applica-
tions were matched to a unique student (on the basis of identiﬁable information) dans le
NCERDC system. A match rate of 100 percent was not expected, since some students
who attend independent schools (private and religious) in Durham County participate
in StudentU. These students are not included in the administrative data at NCERDC
and therefore could not be matched. The overall match rate was 92 percent for lottery
winners and 84 percent for lottery losers, although the pattern of obtaining a higher
match rate among lottery winners was not consistent across cohorts. To assess whether
treatment status diﬀerentially predicted matching, separately by cohort, I regressed a
binary indicator for matching (1 = yes) on a binary indicator for treatment. The diﬀer-
ences in match rates for the 2012, 2013, et 2014 cohorts were 11.4 percentage points
(p = 0.04), −0.6 percentage point (p = 0.92), et 10.9 percentage points (p = 0.12),
respectively.10

Student-Level Data from NCERDC
Administrative data containing information on student characteristics and outcomes
came from NCERDC. These data are provided to NCERDC by the North Carolina De-
partment of Public Instruction and cover the universe of students enrolled in public
and charter schools in North Carolina.

The administrative NCERDC data provided baseline student characteristics and
post-lottery outcomes. Time-invariant demographics included student gender and
race/ethnicity, as well as the following characteristics measured at baseline prior to the
lottery (ﬁfth grade): economically disadvantaged,11 âge, special education status, gifted
designation, English learner (EL) status, and math and reading test scores.

Post-lottery outcomes in early high school included measures constructed from stu-
dent transcripts and summary measures constructed from the universe of reported dis-
ciplinary infractions and their associated consequences (par exemple., suspensions). Outcomes
constructed from raw transcript data included the number of course credits earned
by the end of ninth grade and grade point average.12 Summary measures constructed
from the universe of reported disciplinary infractions included the following binary
indicators: any suspension in ninth grade, any violent or weapons-related disciplinary
infraction, and any other disciplinary infraction.13

Descriptive Statistics
To explore patterns of selection into the StudentU applicant pool, table 1 presents
descriptive statistics for ﬁfth grade students enrolled in traditional public and char-
ter elementary schools in Durham County, North Carolina and matched, non-sibling

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

10. See panel D of table A1.
11.

12.

In this paper, I deﬁne students as economically disadvantaged if they are eligible for free or reduced-price
lunch.
I constructed a weighted version of student grade point average using letter grades and course designations
(Honors, Advanced Placement, etc.) to ensure that GPA was measured consistently across schools. For more
detail on this measure, see online Appendix B.

13. Broadly speaking, the category of all other disciplinary infractions included drug-related, disruptive, sexual,

and miscellaneous oﬀenses. For a sample list of infractions, see online Appendix B.

588

Sarah Komisarow

Tableau 1. Descriptive Statistics, Fifth-Grade Students in Durham County, 2011—12 to 2013—14

(1)

(2)

(3)

Durham

DPS

Charter

0.495
(0.500)
0.500
(0.500)
0.224
(0.417)
0.211
(0.408)
0.173
(0.378)
0.237
(0.425)
0.118
(0.322)
10.524
(0.464)
0.625
(0.484)
0.129
(0.335)
−0.224
(1.042)
−0.241
(1.082)
8,576

0.494
(0.500)
0.499
(0.500)
0.245
(0.430)
0.190
(0.392)
0.171
(0.377)
0.264
(0.441)
0.129
(0.336)
10.530
(0.464)
0.659
(0.474)
0.000
(0.000)
−0.221
(1.048)
−0.264
(1.083)
7,473

0.496
(0.500)
0.512
(0.500)
0.078
(0.268)
0.354
(0.478)
0.182
(0.386)
0.053
(0.224)
0.037
(0.190)
10.484
(0.460)
0.393
(0.489)
1.000
(0.000)
−0.244
(1.005)
−0.090
(1.068)
1,103

(4)

Durham
ED

0.493
(0.500)
0.578
(0.494)
0.319
(0.466)
0.059
(0.235)
0.189
(0.392)
0.149
(0.357)
0.167
(0.373)
10.568
(0.490)
1.000
(0.000)
0.081
(0.273)
−0.521
(0.927)
−0.603
(0.966)
5,328

(5)

DPS
ED

0.492
(0.500)
0.559
(0.497)
0.335
(0.472)
0.061
(0.239)
0.190
(0.392)
0.162
(0.368)
0.176
(0.381)
10.568
(0.486)
1.000
(0.000)
0.000
(0.000)
−0.501
(0.930)
−0.601
(0.966)
4,896

(6)

Charter
ED

0.505
(0.501)
0.792
(0.407)
0.137
(0.344)
0.037
(0.189)
0.178
(0.383)
0.007
(0.083)
0.065
(0.246)
10.569
(0.528)
1.000
(0.000)
1.000
(0.000)
−0.746
(0.866)
−0.626
(0.965)
432

(7)

StudentU
Applicants

(8)

Diff.:
(7)—(1)

0.583
(0.494)
0.478
(0.500)
0.432
(0.496)
0.025
(0.155)
0.142
(0.350)
0.207
(0.406)
0.167
(0.373)
10.509
(0.484)
0.833
(0.373)
0.127
(0.333)
−0.192
(0.863)
−0.338
(0.911)
324

0.089***
(0.028)
−0.022
(0.028)
0.208***
(0.028)
−0.186***
(0.010)
−0.031
(0.020)
−0.030
(0.023)
0.049**
(0.021)
−0.016
(0.027)
0.208***
(0.021)
−0.002
(0.019)
0.032
(0.050)
−0.096*
(0.053)

(9)

Diff.:
(7)—(4)

0.090***
(0.028)
−0.099***
(0.029)
0.113***
(0.028)
−0.034***
(0.009)
−0.047**
(0.020)
0.057**
(0.023)
−0.001
(0.021)
−0.059**
(0.028)
−0.167***
(0.021)
0.045**
(0.019)
0.330***
(0.051)
0.266***
(0.054)

Femelle

Noir

Hispanic

Blanc

Special Ed.

Gifted

Age

Econ. Disad.

Charter

Math Ach.a

Reading Ach.b

Remarques: Columns 1—3 report means and standard deviations (in parentheses) for the variable listed in each row. Column 1 reports descriptive
statistics for 5th-grade students in Durham, Caroline du Nord (Durham Public Schools and charter schools in Durham County) pooled across the 2011—12
to 2013—14 school years. Columns 2 et 3 partition 5th-grade students in Durham County into students in the Durham Public Schools (DPS)
and students enrolled in charter schools in Durham County. Columns 4—6 present information for the same three groups as in columns 1—3 but
restricted to the subsample of students who were economically disadvantaged. Column 7 reports descriptive statistics for StudentU Applicants
who applied for admission in the spring semesters of 2012—14. Column 8 reports the difference in means and associated standard error (dans
parentheses) entre

Columns 1 et 7. Column 9 reports the difference in means and associated standard error (in parentheses) between columns 4 et 7. Special
Ed. = special education; EL = English learner; Econ. Disad. = economically disadvantaged; Ach. = achievement.
aN = 311 for math achievement.
bN = 310 for reading achievement.

StudentU applicants (N = 324; henceforth, I refer to these students as the “matched
sample”). Column 1 reports summary statistics for all ﬁfth-grade students in Durham
County, while column 4 presents summary statistics for the subset of ﬁfth-grade stu-
dents who are economically disadvantaged. The subset of students who are economi-
cally disadvantaged approximates the population of students in Durham County who
are eligible to apply for StudentU, although the approximation is imperfect due to the
fact that StudentU requires applicants to be either free or reduced lunch-eligible and/or
a ﬁrst-generation college student not both.14 Column 7 presents summary statistics for
the matched sample of StudentU applicants.15

14. The administrative data do not provide information on parental education nor ﬁrst-generation college status,
so this restriction based on economic disadvantage is the closest approximation that can be implemented in
the administrative data.
I produce a version of this table for the subset of students from the matched sample (N = 279) with non-
missing covariates and non-missing ninth-grade outcome data (the “regression sample”). See online appendix
table A2.

15.

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

589

Comprehensive Support and Student Success

Aside from documenting diﬀerences between the StudentU applicant pool and
other students in Durham County, I provide further context on the educational set-
ting in Durham County by documenting diﬀerences between the traditional public and
charter sectors. Columns 2 et 3 partition the population of ﬁfth graders in Durham
County into those who attend Durham Public Schools and charter schools, respectivement.
Columns 5 et 6 partition the population of economically disadvantaged ﬁfth graders
in Durham County into the same two subgroups. Comparisons of mean characteris-
tics between the traditional public and charter sectors conﬁrm previously documented
diﬀerences.16 On average, the charter sector contains lower shares of students who are
Hispanic, gifted, ELs, and economically disadvantaged, and a larger share of students
who are white. Students attending charter schools also have lower reading achievement
on average.

StudentU applicants diﬀer on observables from other public and charter students
in Durham County. Column 8 presents diﬀerences in means and associated standard
errors for StudentU applicants versus all other ﬁfth grade students in Durham County.
Relative to all other ﬁfth-grade students in Durham County, StudentU applicants are
more likely to be female (8.9 percentage points [pp]), Hispanic (20.8 pp), designated as
ELs (4.9 pp), and economically disadvantaged (20.8 pp), and they are less likely to be
blanc (18.6 pp).

Column 9 presents diﬀerences in means for StudentU applicants and all economi-
cally disadvantaged ﬁfth-grade students in Durham County. StudentU applicants diﬀer
on observables from the population of economically disadvantaged students in Durham
County. Relative to economically disadvantaged students, StudentU applicants are more
likely to be female (9.0 pp), Hispanic (11.3 pp), gifted (5.7 pp), and enrolled in a char-
ter school (4.5 pp). They are less likely to be black (9.9 pp), blanc (3.4 pp), designated
as special education (4.7 pp), and economically disadvantaged (16.7 pp). They are also
slightly younger when starting ﬁfth grade (0.059 année) and have higher baseline test
scores in math and reading (0.330 et 0.277 SD, respectivement) than the population of
economically disadvantaged ﬁfth-grade students as a whole.

Covariate Balance Tests, Data Missingness, and Attrition
Tableau 2 presents results from covariate balance tests designed to compare StudentU
lottery winners and losers in the matched sample (N = 324). Columns 1 et 2 report
summary statistics to describe lottery winners and lottery losers, respectivement, while col-
umn 3 presents the diﬀerence in means between the lottery winners and lottery losers,
conditional on cohort ﬁxed-eﬀects and cohort ﬁxed-eﬀects interacted with gender. Col-
umn 4 reports the p-value for the test of statistical signiﬁcance for the diﬀerence in
moyens.

Becaues lottery oﬀers were allocated randomly within gender subgroups, là
should not be any diﬀerences between the lottery winners and lottery losers, condi-
tional on cohort ﬁxed-eﬀects and cohort ﬁxed-eﬀects interacted with gender. This is
the case with two exceptions: There is imbalance between lottery winners and lottery
losers in terms of the percent of Hispanic students and share of students who are

16. For a more detailed examination of diﬀerences between the traditional public and charter sectors in the state

of North Carolina, see Ladd, Clotfelter, and Holbein (2016).

590

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

Sarah Komisarow

Tableau 2. Covariate Balance Tests

Noir

Hispanic

Blanc

Special education

Gifted

Age

Economically disadvantaged

Charter

Math achievementa

Reading achievementb

p-value (joint F-test)c

Observations

(1)

Lottery
Winners

0.420
(0.496)

0.520
(0.502)

0.030
(0.171)

0.110
(0.314)

0.240
(0.429)

0.180
(0.386)

10.479
(0.362)

0.910
(0.288)

0.130
(0.338)
−0.100
(0.820)
−0.259
(0.864)

(2)

(3)

Lottery Losers

Diff.: Winners
vs. Losers

0.504
(0.501)

0.393
(0.489)

0.022
(0.148)

0.156
(0.364)

0.192
(0.395)

0.161
(0.368)

10.522
(0.529)

0.799
(0.402)

0.125
(0.331)
−0.234
(0.881)
−0.375
(0.931)

−0.085
(0.061)

0.141
(0.061)

0.005
(0.021)
−0.060
(0.042)

0.045
(0.051)

0.023
(0.047)
−0.036
(0.053)

0.118
(0.042)
−0.007
(0.041)

0.130
(0.106)

0.087
(0.114)

100

224

324

(4)

p-value

0.166

0.022

0.823

0.151

0.383

0.618

0.495

0.005

0.867

0.220

0.445

0.178

Remarques: Column 1 reports raw means and standard deviations for StudentU lottery winners who applied for and
received a lottery offer in the spring semesters of 2012—14. Column 2 reports raw means and standard deviations
for StudentU lottery losers who applied for but did not receive a lottery offer in the spring semesters of 2012—
2014. Column 3 reports the difference in means and associated standard errors in parentheses for StudentU
lottery winners versus lottery losers, conditional on cohort fixed-effects and cohort fixed-effects interacted with
genre. Column 4 reports the p-value associated with a two-tailed t-test of the difference in means reported in
Column 3. The sample includes first-time, non-sibling applicants to StudentU in the 2011—12 to 2013—14 school
années. EL = English learner.
aN = 311 for math achievement.
bN = 310 for reading achievement.
cN = 310 for the regression in which covariates are predictors of treatment status.

economically disadvantaged at baseline (ﬁfth grade). Lottery winners are 14.1 percentage
points more likely than lottery losers to be Hispanic and 11.8 percentage points more
likely to be economically disadvantaged. I do not ﬁnd any other statistically signiﬁcant
diﬀerences between the lottery winners and lottery losers, and I cannot reject the null
hypothèse (F-test) that the covariates are jointly zero as predictors of treatment status.17
As a precaution, I control for student characteristics and covariates measured at base-
line in all subsequent analysis. These controls should account for any ways in which
these characteristics may have inﬂuenced student outcomes. To the extent that this

17.

I produce a version of this table for the subset of the matched sample (N = 279) with non-missing covariates
and non-missing ninth grade outcome data (the “regression sample”). The diﬀerences between the lottery
winners and lottery losers are similar, although the diﬀerence in the proportion of Hispanic students between
groups is smaller in magnitude and only marginally statistically signiﬁcant. See online appendix table A3. Le
diﬀerence in the proportion of economically disadvantaged students is slightly larger and remains statistically
signiﬁcant.

D
o
w
n
o
un
d
e
d

F
r
o
m
h

t
t

:
/
/

d
je
r
e
c
t
.

je
t
.

e
d
toi
e
d
p
un
r
t
je
c
e
–
p
d

F
/

1
7
4
5
7
9
2
0
5
4
7
1
8
e
d
p
_
un
_
0
0
3
6
6
p
d

b
oui
g
toi
e
s
t

o
n
0
8
S
e
p
e
m
b
e
r
2
0
2
3

591

Comprehensive Support and Student Success

measured imbalance aﬀects my estimates, cependant, it is likely to bias treatment eﬀect
estimates toward zero, since the treatment group is more economically disadvantaged
than the control group at baseline.

The main threats to the internal validity of my estimates come from data miss-
ingness (c'est à dire., missing baseline covariates) and sample attrition. In this paper, sample
attrition (c'est à dire., missing ninth-grade outcome data) is the result of (1) students exiting
the public/charter system in North Carolina or (2) repeating a grade following the Stu-
dentU lottery but prior to ninth grade. Sample attrition does not result from dropping
out of StudentU or leaving Durham County, as outcomes are tracked so long as the
student remains enrolled in a public or charter school in North Carolina.

Online appendix table A4 summarizes frequencies and rates of non-missing data
by source, overall and separately by lottery group. When considered row by row, le
table illustrates how the “matched sample” (N = 324) becomes the “regression sample”
(N = 279). From the sequence of rows, two facts emerge: D'abord, missing outcome data
account for most of the reduction in sample size, although missing baseline covariates
contribute, aussi. Deuxième, both of these phenomena have larger impacts on lottery losers.
The table also makes clear that (1) missing baseline demographic characteristics are not
a problem (ﬁrst row), (2) the issue of missing baseline test scores is small (4 pour cent) mais
diﬀerent between lottery groups (lottery winners are 4.8 percentage points more likely
to have non-missing baseline test scores than lottery losers), et (3) after considering
both missing baseline covariates and missing ninth-grade outcomes, lottery winners
are around 11.8 percentage points more likely to be in the “regression sample” than
lottery losers (fourth row).

Missing covariates and missing ninth-grade outcomes could bias my treatment ef-
fect estimates if students who have missing covariates or outcomes diﬀer systematically
from those who do not. I explore this issue in detail for lottery losers, since the issue is
very small for lottery winners (N < 10).18 Online appendix table A5 presents the results from balance tests among lottery losers comparing baseline characteristics for students who do and do not end up in the regression sample. Reassuringly, I do not ﬁnd much evidence that these groups diﬀer on observables, except for the percent of students who are white and student age (in the fall of ﬁfth grade). These diﬀerences, although sta- tistically signiﬁcant, are not practically meaningful. Given the small share of students in the StudentU applicant pool who are white (2.5 percent), this 2.7 percentage point diﬀerence between groups is not unexpected. The 0.36-year age diﬀerence (4 months) is also not practically relevant. The results from these balance tests alleviate some con- cerns about lottery losers either (i) attriting due to lottery status (e.g., discouragement or failing to make on-time grade progress)19 or (ii) compensating for lottery status (e.g., ex- iting the public/charter system for private school), although it is possible that these two eﬀects cancel each other out. As ﬁnal reassurance that diﬀerentially missing outcomes are not responsible for my ﬁndings, I augment presentation of main results with results from three standard methods for dealing with missing outcome data: mean imputation, inverse probability weighting, and multiple imputation. Reassuringly, the main results 18. My Data Use Agreement with NCERDC prohibits the reporting of statistical calculations on groups of students smaller than N = 10. I explore on-time grade progress (a potential eﬀect of StudentU) more fully in section 6. 19. 592 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . / f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow are very similar to those obtained when using these methods. I present the results in online appendix tables D1–D9. 4 . E M P I R I C A L S T R AT E G Y To estimate the eﬀect of a StudentU lottery oﬀer and StudentU enrollment on student outcomes in early high school, I implement a two-stage least-squares (2SLS) approach that exploits the program’s lottery-based admissions system. Using student-level infor- mation on lottery oﬀers combined with information on program take-up, I obtain both intent-to-treat (ITT) and treatment-on-the-treated (TOT) estimates that capture the ef- fects of lottery oﬀers and enrollment, respectively. Below I present my estimating equa- tions, discuss key threats to validity, and report ﬁrst-stage results. Estimating Equations To estimate the reduced-form eﬀect of a StudentU lottery oﬀer on student outcomes, I use an equation of the following form: (cid:2) (cid:2) Yic = α0 + α1 × SUic + γ jdi j + λ jdi j × Femalei + (cid:5)(cid:2) Xic + εic. (1) j j In equation 1, Yic is an outcome (e.g., the number of course credits earned by the end of ninth grade) for student i in cohort c. SUic is a binary variable equal to one if student i won the lottery in cohort c. di j are cohort-speciﬁc dummies, which are included on their own and interacted with gender (j = 2012, 2013, 2014). The interacted dummies account for the fact that StudentU conducted annual admissions lotteries separately for male and female students. Xic is a vector of student-level covariates obtained at baseline (ﬁfth grade), including: gender, race/ethnicity (black, Hispanic, other), charter enrollment, economically disadvantaged, age, gifted designation, special education placement, and EL status. All baseline covariates were measured during the student’s ﬁfth-grade school year, prior to the time when the student learned about whether he or she had received a StudentU lottery oﬀer. The error term, εic, is assumed to be uncorrelated with all other determinants of the outcome and captures random ﬂuctuations in student-level outcomes. To estimate the eﬀect of StudentU enrollment on student outcomes, I implement a 2SLS approach using the following two equations: Enrollic = κ0 + κ1 × SUic + (cid:2) σ jdi j + j Yic = β0 + β1 × (cid:2) Enrollic + (cid:2) j μ jdi j + (cid:2) j ι jdi j × Femalei + (cid:10)(cid:2) Xic + νic. (cid:2) j ζ jdi j × Femalei + (cid:14)(cid:2) Xic + ηic. (2) (3) Equation 2 captures the eﬀect of a lottery oﬀer, SUic, on enrollment in StudentU, Enrollic, for student i in cohort c, conditional on the same covariates, cohort dummies, and cohort dummies interacted with gender as in equation 1. From this ﬁrst-stage (cid:2) Enrollic, and equation I obtained predicted values of the binary enrollment indicator, included these on the right-hand side in the second stage, which is summarized by equation 3. Second-stage estimates of β1 can be interpreted as the average eﬀect of l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / f . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 593 Comprehensive Support and Student Success enrollment in StudentU on student outcomes among the compliers (i.e., those students induced to enroll in StudentU by the lottery oﬀer). This eﬀect is equal to the ratio of α1, the eﬀect of receiving a lottery oﬀer on student outcomes (numerator), toκ1, the eﬀect of receiving a lottery oﬀer on the likelihood of enrollment in StudentU (denominator). First-Stage Results Table A6 in the online appendix reports ﬁrst-stage estimation results from equation 2. The results in column 1 indicate that StudentU lottery winners were around 85.7 percentage points more likely to enroll in StudentU than lottery losers. Enrollment is deﬁned here like “take-up,” reﬂecting any amount of participation in formal StudentU programming (i.e., Summer Academy or the School Year Program).20 The results in columns 2 and 3 demonstrate that addition of student-level demographic covariates and baseline test scores have minimal eﬀects on the magnitude of this point estimate, as is expected if lottery winners and losers are balanced on observables at baseline. Columns 6 and 7 present results separately by baseline achievement test scores.21 Low- achieving lottery winners are slightly more likely to take up the oﬀer of enrollment than high achieving lottery winners (88.0 percent take-up versus 86.6 percent take-up). 5 . M A I N R E S U LT S The Effect of StudentU on Credits Earned in Early High School Panel A of table 3 presents ITT estimates representing the eﬀect of a lottery oﬀer on the number of credits earned by the end of ninth grade. The point estimate in column 3 indicates that lottery winners accumulated around 0.45 more course credits than lottery losers by the end of ninth grade. This translates into a 6 percent increase relative to the control group mean of 7.04 credits. This point estimate is nearly identical to the point estimates presented in columns 1 and 2, which come from speciﬁcations that exclude baseline student covariates and test scores, respectively. Panel B presents TOT results for the same credit accumulation outcome. The TOT estimate represents the eﬀect of enrolling in StudentU on credits earned in ninth grade among the compliers. Due to high take-up rates, the TOT results are very similar to the ITT estimates. To investigate treatment heterogeneity on the basis of prior achievement, I next present results from a speciﬁcation in which lottery status is interacted with an index of baseline student achievement. This index of baseline achievement is an average of each student’s standardized math and reading test scores from ﬁfth grade. I divided the sample into two groups: low and high baseline achievement, respectively, based on the sample median (−0.26 SD). Students in the high baseline achievement group have an average index score of 0.38 SD (median = 0.26 SD), while students in the low baseline achievement group have an average index score of −0.90 SD (median = −0.80 SD). Columns 4 and 5 present results from the interacted model and reveal clear evidence of heterogeneous treatment eﬀects. The point estimate for lottery winners with low baseline achievement is 0.822 credits and is statistically signiﬁcant at the p = 0.05 20. This measure of enrollment in StudentU allows for the possibility that students did not complete the program 21. or, in a few cases, started the program late. I construct an index of baseline achievement by taking the mean of the student’s ﬁfth-grade math and reading test scores. Subject-speciﬁc test scores are expressed in standard deviation units (normalized relative to the statewide mean and standard deviation by grade, subject, and year. 594 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . f / / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d f . / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow Table 3. The Effects of StudentU on Credits Earned in Early High School All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) Panel A. ITT Won lottery Panel B. TOT Enrolled in SU 0.463*** (0.159) 0.480*** (0.160) 0.452*** (0.161) 0.822*** (0.209) 0.082 (0.191) 0.003 0.541*** (0.183) 0.558*** (0.182) 0.524*** (0.181) 0.940*** (0.231) 0.096 (0.218) 0.003 Demographic controls Baseline achievement No No Control mean Observations Yes No 7.04 279 Yes Yes Yes Yes 6.68 143 Yes Yes 7.45 136 Notes: The dependent variable is the number of credits earned at the end of ninth grade. The sample is composed of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Panel A reports intent-to-treat (ITT) estimates, and panel B reports treatment-on-the-treated (TOT) estimates. First-stage estimates are presented in online appendix table A6. Robust standard errors are reported in parentheses. SU = StudentU. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. level, while the point estimate for lottery winners with high baseline achievement is 0.082 credits and is statistically indistinguishable from zero. Column 5 reports the p-value from a statistical test of the diﬀerence in these two coeﬃcients, which allows me to reject the null hypothesis of equality (p = 0.003). Once again, panel B presents TOT estimates that are qualitatively very similar to the ITT estimates. The Effect of StudentU on Grade Point Average in Early High School Panel A of table 4 presents ITT estimates representing the eﬀect of a lottery oﬀer on student grade point average at the end of ninth grade. The point estimates for the full sample in columns 1–3 are small in magnitude and statistically insigniﬁcant. Columns 4 and 5 present results from a speciﬁcation in which lottery status is interacted with an index of baseline student achievement and once again reveal evidence of signiﬁ- cant treatment heterogeneity. The point estimate for lottery winners with low baseline achievement is 0.370 weighted grade points and is statistically signiﬁcant at the 0.05 level. In contrast, the point estimate for lottery winners with high baseline achievement is small, slightly negative, and statistically indistinguishable from zero. The p-value from a statistical test of the diﬀerence in these two coeﬃcients in column 5 is 0.027, which allows me to once again reject the null hypothesis of equality. Panel B presents TOT estimates, which are again qualitatively similar. The Effect of StudentU on the Probability of Suspension in Early High School Panel A of table 5 presents estimates of the eﬀect of a lottery oﬀer on the probability of suspension during ninth grade. The point estimate for the full sample in column 3 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d f / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 595 Comprehensive Support and Student Success Table 4. The Effects of StudentU on Grade Point Average (GPA) in Early High School All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) 0.166 (0.141) 0.176 (0.128) 0.140 (0.120) 0.370*** (0.142) −0.090 (0.171) 0.194 (0.161) 0.205 (0.144) 0.162 (0.135) 0.423*** (0.157) −0.106 (0.195) 0.027 0.023 Panel A. ITT Won lottery Panel B. TOT Enrolled in SU Demographic controls Baseline achievement No No Control mean Observations Yes Yes Yes No 2.81 279 Yes Yes 2.31 143 Yes Yes 3.37 136 Notes: The dependent variable is grade point average at the end of ninth grade. The sample is com- posed of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the be- ginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed- effects interacted with student gender. Panel A reports intent-to-treat (ITT) estimates, and panel B re- ports treatment-on-the-treated (TOT) estimates. First-stage estimates are presented in online appendix table A6. Robust standard errors are reported in parentheses. SU = StudentU. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. Table 5. The Effects of StudentU on the Probability of Suspension in Early High School All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) Panel A. ITT Won lottery Panel B. TOT Enrolled in SU −0.099** (0.047) −0.112** (0.046) −0.103** (0.046) −0.171*** (0.059) −0.036 (0.063) −0.115** (0.054) −0.130** (0.052) −0.120** (0.052) −0.196*** (0.065) −0.042 (0.072) 0.092 0.087 Demographic controls Baseline achievement No No Control mean Observations Yes No .237 279 Yes Yes Yes Yes .306 143 Yes Yes .160 136 Notes: The dependent variable is an indicator that takes on the value of one if the student received at least one suspension in ninth grade. The sample is composed of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English Learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Panel A reports intent-to-treat (ITT) estimates, and panel B reports treatment-on-the-treated (TOT) estimates. First-stage estimates are presented in online appendix table A6. Robust standard errors are reported in parentheses. SU = StudentU. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. 596 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / . f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow indicates that lottery winners were around 10.3 percentage points less likely to be sus- pended during ninth grade than their lottery loser counterparts. In relative terms, this translates into a 47 percent reduction (control mean is 23.7). The point estimates in columns 4 and 5 reveal signiﬁcant evidence of treatment eﬀect heterogeneity on the basis of baseline achievement. The point estimate in column 4 reports a statistically sig- niﬁcant 17.1 percentage point reduction in the likelihood of suspension in ninth grade among lottery winners with low baseline achievement. In contrast, column 4 reports a 3.6 percentage point reduction that is statistically insigniﬁcant. The p-value from a statistical test of the null hypothesis of equality is 0.092 (marginally signiﬁcant). The results from the interacted model once again provide strong evidence to suggest that the mean impact in the full sample of lottery winners is driven by eﬀects on the subgroup of students with low baseline achievement. Discussion of Main Results I ﬁnd substantial evidence to suggest that the eﬀects of StudentU on credit accumula- tion, GPA, and suspension were concentrated among lottery winners who entered the program with low baseline achievement, as measured by ﬁfth-grade math and reading test scores. By the end of ninth grade, these students, who entered the program with average ﬁfth-grade math and reading test scores nearly one standard deviation below the statewide mean (mean = −0.90 SD and median = −0.80 SD among this sub- group), more closely resembled—and in some cases had better outcomes than—their control group (lottery loser) counterparts who entered the program with high baseline achievement. In the cases of credits earned and reduced probability of suspension, lot- tery winners with low baseline achievement had average outcomes that exceeded the average outcomes of their control group counterparts who entered the program with high baseline achievement. In the case of grade point average, the reduced-form treat- ment eﬀect was large enough to close around 35 percent of the gap between low baseline and high baseline achievers in the control group. In contrast to Oreopoulos, Brown, and Lavecchia (2017), who report that the eﬀects of the Pathways program are largest among students with high baseline achievement, this paper documents substantial and consistent evidence of eﬀects concentrated among students with low achievement at baseline. Heterogeneous treatment eﬀects by prior achievement are not reported in Rodriguez-Planas (2012), although she does report signiﬁcant evidence of heteroge- neous treatment eﬀects by gender.22 Taken together with previous work, these ﬁndings suggest that comprehensive services delivered outside of the regular school day in a uniﬁed setting may be most eﬀective at improving educational outcomes among stu- dents with low levels of academic achievement at baseline. Extra time outside of school in a small-group and structured setting may improve study skills, provide students with the support needed to keep up with regular assignments and homework, and oﬀer op- portunities to ask questions about assigned work. These ﬁndings are consistent with results reported in Kraft (2015), who ﬁnds that low-achieving students beneﬁt more than high-achieving students from small-group tutorials that oﬀer individualized and 22. I investigate the possibility of heterogeneous treatment eﬀects by gender but do not ﬁnd any evidence statisti- cally diﬀerent treatment eﬀects for male versus female students. The results are reported in online appendix tables C1–C3. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / f / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . f / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 597 Comprehensive Support and Student Success diﬀerentiated instruction outside of the regular school day. The ﬁndings are also con- sistent with experimental evidence from Schueler (2020), who ﬁnds that small-group academic instruction in math—even in small weeklong doses—provided by a single teacher led to an increased likelihood that students were proﬁcient in math and to less exclusionary discipline. StudentU delivers all of the components of its programming to students simultane- ously, which makes it impossible—in this setting—to identify how speciﬁc parts of the program (e.g., small group instruction, small-group mentoring, nutrition, child/family services) aﬀected student outcomes. It also makes it impossible to identify whether there were complementarities between the program’s components that made them more eﬀective because they were delivered together rather than in isolation. These questions, although unanswered here, are important and deserve additional attention in future research. Despite these limitations, however, it is worth noting that the struc- ture of StudentU programming shares several features identiﬁed in previous work as critical to the success of out-of-school time interventions. Among other factors, Schwartz et al. (2018) identify “high-quality enrichment experiences” and “a high level of engagement between adults and students” as factors crucial to the success of out of school time programs. 6 . M E C H A N I S M S To gain insight into the mechanisms underlying increased credit accumulation, higher grade point averages, and reduced likelihood of suspension among lottery winners with low baseline achievement, I investigated several intervening variables that provide in- sight into the causal channels through which StudentU aﬀects student outcomes. To distinguish between channels of on-time grade progress, course-taking and comple- tion, achievement (cognitive skills), and behavior, I reestimated the same models for several intervening variables that serve as proxies for these channels. Table 6 presents ITT estimates of the eﬀect of a lottery oﬀer on on-time grade progress following admission to StudentU (i.e., for sixth through ninth grades). The point estimate in column 3 indicates that lottery winners were around 2.1 percentage points (2 percent) more likely to make on-time grade progress than lottery losers, al- though the eﬀect is only marginally statistically signiﬁcant. The point estimate for lot- tery winners with low baseline achievement is 3.2 percentage points and statistically signiﬁcant, while the point estimate for lottery winners with high baseline achieve- ment is 0.1 percentage point and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.133). For completeness, I present TOT estimates for the same outcomes in online appendix table A8. Table 7 presents ITT estimates representing the eﬀect of a lottery oﬀer on credits at- tempted in ninth grade and a binary variable indicating whether the student failed any courses during ninth grade. The point estimates in panel A do not reveal and statisti- cally signiﬁcant eﬀects on credits attempted, although the point estimates in panel B do reveal substantial reductions in the likelihood of any course failure. The point estimate in column 3 indicates that lottery winners in the full sample are around 8.6 percent- age points less likely to fail any courses relative to their lottery loser counterparts. The point estimates in columns 4 and 5 make clear that this eﬀect is entirely driven by lot- tery winners with low baseline achievement, who are 20.4 percentage points less likely 598 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . / f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow Table 6. The Effects of StudentU on On-Time Grade Progress (Intent-to-Treat) All Students Baseline Achievement (1) (2) (3) Low (4) On-time grade progression Control mean Observations Demographic controls Baseline achievement 0.027** (0.012) 0.975 295 No No 0.027** (0.012) 0.975 0.021* (0.011) 0.979 0.032** (0.016) 0.961 295 Yes No 283 Yes Yes 136 Yes Yes p-value (6) 0.133 High (5) 0.010 (0.009) 1.000 147 Yes Yes Notes: The dependent variable is a binary indicator for on-time grade progress through ninth grade (0/1). The sample is comprised of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Robust standard errors are reported in parentheses. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. Table 7. The Effects of StudentU on Student Course-Taking and Completion (Intent-to-Treat) All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) Panel A. Credits attempted Control mean Observations Panel B. Course failure (0/1) Control mean Observations Demographic controls Baseline achievement 0.132* (0.074) 7.692 279 0.130* (0.074) 7.692 279 0.119 (0.074) 7.692 279 0.167 (0.102) 7.633 136 −0.081 (0.053) −0.096* (0.052) −0.086* (0.051) −0.204*** (0.069) 0.265 279 No No 0.265 0.265 0.398 279 Yes No 279 Yes Yes 136 Yes Yes 0.072 (0.094) 7.759 143 0.031 (0.065) 0.115 143 Yes Yes 0.464 0.008 Notes: The dependent variable in panel A is the number of credits attempted during ninth grade, and the dependent variable in panel B is an indicator that takes on the value of one if the student received at least one failing course grade during ninth grade. The sample is composed of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Robust standard errors are reported in parentheses. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. to fail any courses during ninth grade. This contrasts with the small and statistically insigniﬁcant point estimate among lottery winners with high baseline achievement. The p-value from a statistical test of the diﬀerence between these coeﬃcients is p = 0.008, which means that I can reject the null hypothesis of equality. For completeness, l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / f / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d f / . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 599 Comprehensive Support and Student Success Table 8. The Effects of StudentU on Student Achievement (Eighth-Grade Test Scores) (Intent-to-Treat) All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) Panel A. Math achievement Control mean Observations Panel B. Reading achievement Control mean Observations Demographic controls Baseline achievement 0.029 (0.124) −0.199 275 0.104 (0.115) −0.205 275 No No −0.010 (0.105) −0.199 275 0.063 (0.096) −0.205 275 Yes No −0.043 (0.089) −0.199 275 0.018 (0.079) −0.205 275 Yes Yes 0.071 (0.114) −0.672 135 0.103 (0.105) −0.673 135 Yes Yes −0.154 (0.121) 0.329 140 −0.065 (0.097) 0.319 140 Yes Yes 0.147 0.187 Notes: The dependent variable in panel A is math achievement in ninth grade (measured in standard deviation units), and the dependent variable in panel B is reading achievement in ninth grade (measured in standard deviation units). The sample is composed of the 2012—14 StudentU cohorts (lottery winners and losers). Spec- ifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Robust standard errors are reported in parentheses. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. I present TOT estimates for the same outcomes in online appendix table A9. These results are qualitatively similar to the reduced-form eﬀects. I do not ﬁnd any statistically signiﬁcant evidence to suggest that improvements in cognitive skills are a likely mechanism through which lottery winners improve their early high school outcomes, although I acknowledge that low statistical power is a con- cern for these outcomes.23 Table 8 presents ITT estimates of the eﬀect of winning the lottery on math and reading test scores in eighth grade.24 For completeness, I present TOT estimates for the same outcomes in online appendix table A10. Table 9 presents ITT estimates representing the eﬀect of a lottery oﬀer on the like- lihood of a violent/weapons-related and other disciplinary infractions in ninth grade. Although I do not ﬁnd any evidence of reductions in the likelihood of violent/weapons- related infractions, I ﬁnd substantial evidence of reductions in the likelihood of other disciplinary infractions in panel B. The point estimates in column 4 indicate that lot- tery winners in the full sample are around 10.3 percentage points less likely to have a reported other infraction in ninth grade. The point estimates in columns 4 and 5 reveal that this main eﬀect is likely driven by lottery winners with low baseline achievement. The point estimate for lottery winners with low baseline achievement is 16.1 percent- age points and statistically signiﬁcant, while the point estimate for lottery winners with I estimate that the minimum detectable eﬀect is 0.2 SD for test scores in both math and reading. 23. 24. There are no achievement test scores available in ninth grade. 600 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / f / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / f . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow Table 9. The Effects of StudentU on Student Behavior (Intent-to-Treat) All Students Baseline Achievement (1) (2) (3) Low (4) High (5) p-value (6) −0.038 (0.034) 0.097 279 −0.036 (0.034) 0.097 279 −0.032 (0.035) 0.097 279 −0.021 (0.047) 0.112 136 −0.100** (0.046) −0.113** (0.045) −0.103** (0.045) −0.161*** (0.059) 0.227 279 No No 0.227 0.227 0.296 279 Yes No 279 Yes Yes 136 Yes Yes −0.042 (0.040) 0.080 143 −0.046 (0.060) 0.149 143 Yes Yes 0.692 0.146 Panel A. Violent/weapons infractions Control mean Observations Panel B. Other infractions Control mean Observations Demographic controls Baseline achievement Notes: The dependent variable in panel A is an indicator that takes on the value of one if the student was reported for at least one violent/weapons-related disciplinary infraction in ninth grade, and the dependent variable in panel B is an indicator that takes on the value of one if the student was reported for at least one other disciplinary infraction in ninth grade. The sample is comprised of the 2012—14 StudentU cohorts (lottery winners and losers). Specifications with demographic controls include female, black, Hispanic, economically disadvantaged in fifth grade, age (at the beginning of fifth grade), charter enrollment in fifth grade, gifted in fifth grade, special education in fifth grade, and English learner in fifth grade. Specifications with baseline achievement controls include math achievement in fifth grade and reading achievement in fifth grade. Models estimated on the full sample and by separately baseline achievement include cohort fixed-effects and cohort fixed-effects interacted with student gender. Robust standard errors are reported in parentheses. Asterisks denote statistical significance: *p < 0.10, **p < 0.05, ***p < 0.01. high baseline achievement is 4.6 percentage points and statistically indistinguishable from zero, although I cannot reject the null hypothesis of equality (p = 0.146). Taken together, the results suggest that StudentU improved course completion and reduced disciplinary infractions in school among lottery winners with low baseline achievement. I do not ﬁnd any evidence of any improvements in student achievement. This is consistent with previous literature on the eﬀects of tutoring outside of the regu- lar school day (Heinrich, Meyer, and Whitten 2010; Zimmer, Hamilton, and Christina 2010; Heinrich et al. 2014). It contrasts with work that reports results from one-on- one or two-on-one, high-dosage tutoring that occurs within the context of the regular school day (see, e.g., Fryer 2014; Kraft 2015; Ander, Guryan, and Ludwig 2016; Chabrier, Cohodes, and Oreopoulos 2016). 7 . P R E D I C T E D E F F E C T O N H I G H S C H O O L G R A D UAT I O N A N D P RO G R A M C O S T S To assess the likely eﬀects of improved early high school outcomes on high school grad- uation, I constructed an index based on students’ early high school outcomes and esti- mated the empirical relationship between this index and the likelihood of subsequent high school graduation. The binary index (henceforth, the Early High School Outcomes Index) was equal to one if at the end of ninth grade a student (1) had earned the mini- mum number of credits required to transition to tenth grade and (2) had not received any suspensions. The Early High School Outcomes Index was equal to zero otherwise. I then estimated the relationship between the Early High School Outcomes Index and the l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / . f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 601 Comprehensive Support and Student Success likelihood that a student graduated from high school within ﬁve years using data from three successive cohorts of ﬁrst-time ninth graders in Durham County (this includes students in traditional public and charter schools). Although this relationship was es- timated among cohorts of ﬁrst-time ninth graders in Durham County, the statewide longitudinal data permit me to track high school graduation from any public or charter school in the state of North Carolina. Figure 2a plots the empirical relationship between the Early High School Outcomes Index and the likelihood of graduation from high school within ﬁve years. A one-unit change in the Early High School Outcomes Index is associated with a 29 percentage point increase in the likelihood that a student graduates from high school within ﬁve years (control mean = 0.59). Although there is some variation across cohorts, the rela- tionship appears to be quite stable. Figure 2b depicts the eﬀect of winning the StudentU lottery on the likelihood that the Early High School Outcomes Index is equal to one. The pooled estimate indicates that StudentU lottery winners were around 12 percentage points more likely to have an Early High School Outcomes Index equal to one when compared with their lottery loser counterparts (control mean = 0.51). Based on these estimated relationships, I calculate an expected graduation rate of 77 percent for lottery winners and 73 percent for lottery losers, which implies that the eﬀect of winning the lottery on the likelihood of high school graduation is a 4 percentage point increase. This predicted eﬀect on high school graduation is considerably smaller than the actual eﬀect realized in the Pathways program at the original site of implementation (15.3 percentage points), although it is similar in magnitude to the Pathways eﬀect at the expansion site (5.8 percentage points) and to the (statistically insigniﬁcant) eﬀect reported for the QOP (3.2 percentage points) (Oreopoulos, Brown, and Lavecchia 2017; Rodriguez-Planas 2012). The predicted eﬀect of StudentU on high school graduation is large and meaningful but should not be considered in isolation. When evaluating any educational interven- tion, it is important to consider not only the eﬀects—and their social value—but also the level of investment required to obtain said impacts. To obtain estimates of per student spending for StudentU, I used three years’ worth of annual expenditure data broken down by program and one year’s oﬃcial tax records that reported expenditures broken down by category (e.g., salary/wages, insurance, supplies, food).25 Using these data, I estimated per student spending between $4,790 and $5,905 (2016 dollars) per year. Summing these costs over four years and assuming a real discount rate of 7 percent, I obtained a total cost estimate of $16,225 to $20,002 (2016 dollars) per student, which represents the total cost per student for four years of StudentU programming and ser- vices (this corresponds to the number of years between a student’s matriculation into the program and the early high school outcomes observed in this paper). This range of cost estimates compares favorably to estimates from other comprehensive programs and related programs in the literature. Oreopoulos, Brown, and Lavecchia (2017) re- port a per student cost estimate of $13,400 for Pathways and Rodriguez-Planas (2012) reports a per student cost estimate of $25,000 for QOP (4-year program). 25. For a more detailed explanation of the expenditure data and my calculations, see online Appendix B. 602 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / f / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / . f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . f / / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d f . / Notes: (a) Coefficient estimates for the effect of the Early High School Outcomes Index on the likelihood of high school graduation (within 5 years) for three successive cohorts of first-time ninth-grade students in Durham County, NC. (b) Coefficient estimates for the effect of winning the StudentU lottery on the probability that the Early High School Outcomes Index is equal to one. Figure 2. Predicted Effect of StudentU (SU) on High School Graduation 8 . C O N C L U S I O N In this paper I investigate the eﬀects of StudentU, a comprehensive, multiyear program designed to improve educational outcomes among disadvantaged students, on the early high school outcomes of participating students. To do this, I exploit variation induced f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 603 Comprehensive Support and Student Success by oﬀers from oversubscribed admissions lotteries. Although I ﬁnd some evidence of eﬀects on early high school outcomes among the full sample of lottery winners, more detailed investigation reveals that these were driven exclusively by the subgroup of lot- tery winners who entered the program with low baseline achievement. In an investigation of the mechanisms underlying these eﬀects, I ﬁnd evidence to suggest that on-time grade progress, course completion, and reduced reports of disci- plinary infractions in school are the likely channels through which these eﬀects ﬂowed. Speciﬁcally, I ﬁnd that lottery winners were around 3.2 percentage points more likely to make on-time grade progress, 20.4 percentage points less likely to fail a course during ninth grade, and around 10.3 percentage points less likely to be reported for a miscel- laneous (i.e., not violent and not weapons-related) disciplinary infraction during ninth grade. As a means to forecast longer-run impacts on students’ high school graduation out- comes, I conclude by presenting the results from a simple prediction exercise designed assess how these improvements in early high school outcomes likely translate into suc- cesses later in high school. Using an index of students’ early high school outcomes, I predict that lottery winners are around 4 percentage points more likely to graduate from high school than lottery losers (5 percent eﬀect). These results are squarely in line with estimated eﬀects presented in the previous literature. I also report estimates of per-student program costs, which also compare favorably to estimates reported for similar comprehensive programs. Taken together, the results from this paper suggest that comprehensive services delivered outside of the regular school day have the po- tential to improve the educational outcomes of disadvantaged students outside of the regular school day. ACKNOWLEDGMENTS I thank the Editors, Randall Reback and Andrew McEachin, and three anonymous referees for their helpful comments and suggestions. I am grateful to Kara Bonneau and Radhika Deshmukh- McDiarmid for their assistance in matching the data for this project. Liz Ananat, Steve Bednar, Barbara Biasi, Sandy Black, Russ Bittmann, Charlie Clotfelter, Sue Dynarski, Matt Johnson, Ezra Karger, Sunny Ladd, Alicia Modestino, Nolan Pope, Seth Sanders, Elizabeth Setren, Crystal Zhan, and participants at the UNC-CH Educational Inequality Workshop, Elon University, the AEFP 44th Annual Conference, the SEA 88th Annual Meeting, the 2019 Triangle Applied Micro Con- ference, and the 2019 Carolina Region Empirical Economics Day provided helpful comments on this draft and on earlier versions of this paper. Graeme Peterson provided exceptional research assistance. All errors are my own. REFERENCES Ander, Roseanna, Jonathan Guryan, and Jens Ludwig. 2016. Improving academic outcomes for disadvantaged students: Scaling up individualized tutorials. Policy Proposal No. 2016-02. The Hamilton Project: The Brookings Institution. Angrist, Joshua, Daniel Lang, and Philip Oreopoulos. 2009. Incentives and services for college achievement: Evidence from a randomized trial. American Economic Journal: Applied Economics 1(1): 136–163. 10.1257/app.1.1.136 Augustine, Catherine H., Jennifer Sloan McCombs, John F. Pane, Heather L. Schwartz, Jonathan Schweig, Andrew McEachin, and Kyle Siler-Evans. 2016. Learning from summer: Eﬀects of 604 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . f / / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . / f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow voluntary summer learning programs on low-income urban youth. RAND Education. Santa Mon- ica, CA: RAND Corporation. Augustine, Catherine H., and Lindsey E. Thompson. 2020. Getting support for summer learn- ing: How federal, state, city, and district policies aﬀect summer learning programs. RAND Edu- cation and Labor. Santa Monica, CA: RAND Corporation. Bettinger, Eric P., and Rachel B. Baker. 2014. The eﬀects of student coaching: An evaluation of a randomized experiment in student advising. Educational Evaluation and Policy Analysis 36(1): 3–19. 10.3102/0162373713500523 Bhattacharya, Jayanta, Janet Currie, and Steven J. Haider. 2006. Breakfast of champions? The school breakfast program and the nutrition of children and families. Journal of Human Resources 41(3): 445–466. 10.3368/jhr.XLI.3.445 Carrell, Scott, and Bruce Sacerdote. 2017. Why do college-going interventions work? American Economic Journal: Applied Economics 9(3): 124–151. 10.1257/app.20150530 Castleman, Benjamin L., Lindsay C. Page, and Korynn Schooley. 2014. The forgotten summer: Does the oﬀer of college counseling after high school mitigate summer melt among college- intending, low-income high school graduates? Journal of Policy Analysis and Management 33(2): 324–334. Chabrier, Julia, Sarah Cohodes, and Philip Oreopoulos. 2016. What can we learn from charter school lotteries? Journal of Economic Perspectives 30(3): 57–84. 10.1257/jep.30.3.57 Clotfelter, Charles T., Steven W. Hemelt, and Helen F. Ladd. 2018. Multifaceted aid for low- income students and college outcomes: Evidence from North Carolina. Economic Inquiry 56(1): 278–303. 10.1111/ecin.12486 Cook, Philip J., Kenneth A. Dodge, George Farkas, Roland G. Fryer, Jonathan Guryan, Jens Lud- wig, Susan Mayer, Harold A. Pollack, and Laurence Steinberg. 2014. The (surprising) eﬃcacy of academic and behavioral intervention with disadvantaged youth: Results from a randomized experiment in Chicago. NBER Working Paper No. 19862. Corcoran, Sean P., Brian Elbel, and Amy Ellen Schwartz. 2016. The eﬀect of breakfast in the classroom on obesity and academic performance: Evidence from New York City. Journal of Policy Analysis and Management 35(3): 509–532. 10.1002/pam.21909 Dobbie, Will, and Roland G. Fryer. 2011. Are high-quality schools enough to increase achievement among the poor? Evidence from the Harlem children’s zone. American Economic Journal: Applied Economics 3(3): 158–187. 10.1257/app.3.3.158 Duncan, Greg J., and Katherine Magnuson. 2011. The nature and impact of early achievement skills, attention skills, and behavior problems. In Whither opportunity: Rising inequality, schools, and children’s life chances, edited by Greg J. Duncan and Richard J. Murnane, pp. 47–70. New York: Russell Sage Foundation. Farkas, George. 2011. Middle and high school skills, behaviors, attitudes, and curriculum enroll- ment, and their consequences. In Whither opportunity: Rising inequality, schools, and children’s life chances, edited by Greg J. Duncan and Richard J. Murnane, pp. 71–90. New York: Russell Sage Foundation. Ford, Reuben, and Isaac Kwakye. 2016. Future to discover: Sixth year post-secondary impacts report. Ottawa, Ontario: Social Research and Demonstration Corporation. l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / f / e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d / f . f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 605 Comprehensive Support and Student Success Fryer, Roland G. 2014. Injecting charter school best practices into traditional public schools: Evi- dence from ﬁeld experiments. Quarterly Journal of Economics 129(3): 1355–1407. 10.1093/qje/qju011 Fryer, Roland G. 2017. The production of human capital in developed countries: Evidence from 196 randomized ﬁeld experiments. In Handbook of economic ﬁeld experiments, edited by Abhijit Vinayak Banerjee and Esther Duﬂo, pp. 295–322. Amsterdam, Netherlands: North-Holland. Gandhi, Allison Gruner, Rachel Slama, So Jung Park, Patrick Russo, Kendra Winner, Robin Bzura, Wehmah Jones, and Sandra Williamson. 2018. Focusing on the whole student: An evalua- tion of Massachusetts’s wraparound zone initiative. Journal of Research on Educational Eﬀectiveness 11(2): 240–266. 10.1080/19345747.2017.1413691 Gassman-Pines, Anna, and Laura Bellows. 2018. Food instability and academic achievement: A quasi-experiment using snap beneﬁt timing. American Educational Research Journal 55(5): 897– 927. 10.3102/0002831218761337 Grossman, Jean Baldwin, and Joseph P. Tierney. 1998. Does mentoring work? An impact study of the big brothers big sisters program. Evaluation Review 22(3): 403–426. 10.1177/ 0193841X9802200304 Heinrich, Carolyn J., Patricia Burch, Annalee Good, Rudy Acosta, Huiping Cheng, Marcus Dil- lender, Christi Kirshbaum, Hiren Nisar, and Mary Stewart. 2014. Improving the implementation and eﬀectiveness of out-of-school-time tutoring. Journal of Policy Analysis and Management 33(2): 471–494. 10.1002/pam.21745 Heinrich, Carolyn J., Robert H. Meyer, and Greg Whitten. 2010. Supplemental education services under no child left behind: Who signs up, and what do they gain? Educational Evaluation and Policy Analysis 32(2): 273–298. 10.3102/0162373710361640 Heller, Sara B., Harold A. Pollack, Roseanna Ander, and Jens Ludwig. 2013. Preventing youth violence and dropout: A randomized ﬁeld experiment. NBER Working Paper No. 19014. Heller, Sara B., Anuj K. Shah, Jonathan Guryan, Jens Ludwig, Sendhil Mullainathan, and Harold A. Pollack. 2017. Thinking, fast and slow? Some ﬁeld experiments to reduce crime and dropout in Chicago. Quarterly Journal of Economics 132(1): 1–54. 10.1093/qje/qjw033 Imberman, Scott A., and Adriana D. Kugler. 2014. The eﬀect of providing breakfast in class on stu- dent performance. Journal of Policy Analysis and Management 33(3): 669–699. 10.1002/pam.21759 Johnston, William R., Celia J. Gomez, Lisa Sontag-Padilla, Lea Xenakis, and Brent Anderson. 2017. Developing community schools at scale: Implementation of the New York City community schools initiative. RAND Corporation. Kim, James S., and David M. Quinn. 2013. The eﬀects of summer reading on low-income chil- dren’s literacy achievement from kindergarten to grade 8: A meta-analysis of classroom and home interventions. Review of Educational Research 83(3): 386–431. 10.3102/0034654313483906 Kraft, Matthew A. 2015. How to make additional time matter: Integrating individualized tutorials into an extended day. Education Finance and Policy 10(1): 81–116. 10.1162/EDFP_a_00152 Ladd, Helen F., Charles T. Clotfelter, and John B. Holbein. 2016. The growing segmentation of the charter school sector in North Carolina. Education Finance and Policy 12(4): 536–563. 10.1162/ edfp_a_00226 Leuven, Edwin, Hessel Oosterbeek, and Bas van der Klaauw. 2010. The eﬀect of ﬁnancial rewards on students’ achievement: Evidence from a randomized experiment. Journal of the European Eco- nomic Association 8(6): 1243–1265. 606 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . f / f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 Sarah Komisarow McCallion, Gail. 2001. 21st century community learning centers: An overview of the program and analysis of reauthorization issues. Order Code RL30306. Washington, DC: Congressional Research Service. McCombs, Jennifer Sloan, Catherine H. Augustine, Fatih Unlu, Kathleen M. Ziol-Guest, Scott Naftel, Celia J. Gomez, Terry Marsh, Goke Akinniranye, and Ivy Todd. 2019. Investing in suc- cessful summer programs: A review of evidence under the Every Student Succeeds Act. RAND Education and Labor. Santa Monica, CA: RAND Corporation. Opper, Issac M., William R. Johnston, John Engberg, and Lea Xenakis. 2019. Assessing the short- term impact of the New York City renewal schools program. Working Paper WR-1303-NYCDOE. RAND Education and Labor. Oreopoulos, Philip, Robert S. Brown, and Adam M. Lavecchia. 2017. Pathways to education: An integrated approach to helping at-risk high school students. Journal of Political Economy 125(4): 947–984. 10.1086/692713 Reardon, Sean F. 2011. The widening academic achievement gap between the rich and the poor: New evidence and possible explanations. In Whither opportunity: rising inequality, schools, and chil- dren’s life chances, edited by Greg J. Duncan and Richard J. Murnane, pp. 91–116. New York: Russell Sage Foundation. Rodriguez-Planas, Nuria. 2012. Longer-term impacts of mentoring, educational services, and learning incentives: Evidence from a randomized trial in the United States. American Economic Journal: Applied Economics 4(4): 121–139. 10.1257/app.4.4.121 Schueler, Beth E. 2020. Making the most of school vacation: A ﬁeld experiment of small group math instruction. Education Finance and Policy 15(2): 310–331. 10.1162/edfp_a_00269 Schwartz, Heather L., Jennifer Sloan McCombs, Catherine H. Augustine, and Jennifer T. Les- chitz. 2018. Getting to work on summer learning: Recommended practices for success, 2nd ed. RAND Education and Labor. Santa Monica, CA: RAND Corporation. Stapleton, Katina R. 1998. 21st century community learning centers: A summary of the program. Report Number CRS-1998-EPW-0113. Washington, DC: Congressional Research Service. Winters, Marcus A. 2017. Costly progress: DeBlasio’s renewal school program. Report. New York: Manhattan Institute. Zimmer, Ron, Laura Hamilton, and Rachel Christina. 2010. After-school tutoring in the context of no child left behind: Eﬀectiveness of two programs in the Pittsburgh public schools. Economics of Education Review 29(1): 18–28. 10.1016/j.econedurev.2009.02.005 l D o w n o a d e d f r o m h t t p : / / d i r e c t . m i t . / / f e d u e d p a r t i c e - p d l f / / / / 1 7 4 5 7 9 2 0 5 4 7 1 8 e d p _ a _ 0 0 3 6 6 p d . / f f b y g u e s t t o n 0 8 S e p e m b e r 2 0 2 3 607 Sarah Komisarow image

Télécharger le PDF