EFFECTS OF FLIPPED CLASSROOM - IA de Investigación especializada en el MIT

EFFECTS OF FLIPPED CLASSROOM

INSTRUCTION: EVIDENCE FROM

A RANDOMIZED TRIAL

Abstracto
In a ﬂipped classroom, an increasingly popular pedagogical
modelo, students view a video lecture at home and work on ex-
ercises with the instructor during class time. Advocates of the
ﬂipped classroom claim the practice not only improves student
achievement but also ameliorates the achievement gap. Nosotros estafamos-
duct a randomized controlled trial at West Point and ﬁnd the
ﬂipped classroom produced short-term gains in math and no ef-
fect in economics. The ﬂipped model broadened the achievement
gap: Eﬀects are driven by white, masculino, and higher-achieving stu-
abolladuras. We ﬁnd no long-term average eﬀects on student learn-
ing but the widened achievement gap persists. Our ﬁndings
demonstrate feasibility for the ﬂipped classroom to induce short-
term gains in student learning; sin embargo, the exacerbation of the
achievement gap, the eﬀect fade-out, and the null eﬀects in eco-
nomics, suggest that educators should exercise caution when con-
sidering the model.

https://doi.org/10.1162/edfp_a_00314

Elizabeth Setren

(Autor correspondiente)

Departamento de Economía

Tufts University

Medford, MAMÁ 02155

Elizabeth.Setren@tufts.edu

Kyle Greenberg

Department of Social Sciences

United States Military Academy

West Point, Nueva York 10996

kyle.greenberg@westpoint.edu

Oliver Moore

Oﬃce of the Deputy Chief

of Staﬀ for Programming

Army G-8, Headquarters

Department of the Army

Washington, corriente continua 20310

oliver.c.moore1@gmail.com

Michael Yankovich

Department of Mathematical

Ciencias

United States Military Academy

West Point, Nueva York 10996

michael.yankovich@westpoint.edu

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

363

Effects of Flipped Classroom Instruction

INTRODUCCIÓN

1 .
Technology plays an increasing role in education and opens up a myriad of possibili-
ties for educators to innovate on the traditional lecture format. One option, called the
“ﬂipped classroom,” involves students learning the material by watching video lectures
prior to class. This frees up class time for more in-depth discussion and application of
the concepts through practice problems, group work, and increased interaction with the
instructor (Brame 2013). Industry surveys estimate that over half of U.S. colleges use
the ﬂipped classroom and its popularity is growing (Schaﬀhuaser 2016; Schaﬀhauser
and Kelly 2016). A range of education nonproﬁts, textbook publishers, and technology
companies have capitalized on increasing interest in the format by providing videos
and other educational tools.

Despite the proliferation of the ﬂipped classroom, little well-identiﬁed evidence
exists on its impact on student learning. Proponents claim this model not only boosts
student achievement but also ameliorates the achievement gap through increased
student–teacher interaction (Supiano 2018). The increased contact with students may
make instructors more responsive to students’ needs, which could be particularly
beneﬁcial for lower-achieving students who might otherwise not seek out assistance
(Bergmann and Sams 2012; Goodwin and Miller 2013).

Opponents of the ﬂipped classroom worry that it requires extensive work by in-
structors to create engaging videos and interactive classroom activities and hinges on
students’ engagement with the material outside of class (Lo and Hew 2017). The ﬂipped
classroom may take more time for students and will leave them without a foundational
overview of the concepts if they do not watch the videos. Además, if lower-income
students have less reliable Internet access, it could exacerbate inequalities.

This study presents causal estimates of the ﬂipped classroom’s impact on student
learning using a randomized controlled trial at West Point. We conducted the study
during one unit in two mandatory core-curriculum courses, Introduction to Calculus
and Principals of Economics, allowing us to explore the impact of the ﬂipped classroom
in two diﬀerent subjects. Aspects of West Point and these two classes make it an ideal
setting for this randomized controlled trial. Both courses require extensive problem-
solving, a common attribute of ﬂipped classroom courses (Berrett 2012). They also lend
themselves well to consistent grading to provide an objective measurement of student
aprendiendo. West Point standardizes the curriculum, teaching, and exams of these two
high-enrollment courses across the eighty course sections. Además, the registrar
randomly assigns students to course sections so that the sections have similar sets of
estudiantes.

We randomly assigned course sections to ﬂipped classrooms or standard lectures.
To remove individual instructor eﬀects on student learning outcomes, we assigned each
instructor to at least one section in the control group and one section in the treatment
grupo. The ﬂipped classroom treatment consisted of a standardized video lecture that
instructors told students to watch before class, and interactive problem-solving during
class time. Students in the control group received a standardized lecture in class, con
identical content to the video lecture. They were also given the same problems to work
on as the treatment group, but to solve outside of class. To isolate the impact of the
ﬂipped format, we held the lecture material, instructor, and practice problems constant,
and altered the format and time in which students engage with the practice problems

364

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

and lectures. If students complete all of the assigned work, we expect students to spend
a similar amount of time preparing for class: completing the readings and watching
the videos for the ﬂipped classroom and completing the readings and working on the
practice problems for the standard lectures.

We ﬁnd that the ﬂipped classroom does not reduce the achievement gap as pro-
ponents suggest. The ﬂipped classroom produced a strong, positive short-term eﬀect
in math and null eﬀect in economics. Students in the ﬂipped math classrooms scored
0.3 standard deviations above the mean on the unit quiz relative to their peers in the
standard classroom. Sin embargo, femenino, negro, and Hispanic students, and students with
lower baseline academic performance (measured by their ACT scores), do not expe-
rience gains from the math ﬂipped classroom; the math eﬀects are driven by white,
masculino, and higher-achieving students. The ﬂipped classroom has a 69 percent larger
white/nonwhite (black or Hispanic) achievement gap relative to the standard lecture
and it exacerbates the diﬀerence by 23 percent between students who scored in the top
and bottom ACT quartile. Although the average eﬀects fade by the course ﬁnal, el
achievement gaps persist. We observe lower levels of student engagement (both in and
out of the classroom) and instructor interest in the less-eﬀective ﬂipped classrooms.
Our ﬁndings demonstrate that it is feasible for the ﬂipped classroom to induce large
gains in student learning in a short period of time, but that the eﬀects vary by subject,
student characteristics, and teacher motivation for the ﬂipped classroom technique.
The exacerbation of the achievement gap, the fade-out of eﬀects, and the diﬀerent ef-
fects by subject suggest that educators should exercise caution when considering the
ﬂipped classroom.

This study contributes to a growing literature on technology and education (ver
Escueta et al. 2017 for a summary). Within the education technology literature, trabajar
on the impact of online courses is particularly relevant. Randomized studies ﬁnd small
negative eﬀects of fully online courses compared to in-person lectures (Figlio, Rush, y
Yin 2013) and similar eﬀects of standard lectures relative to reduced lecture time with
access to online course materials or machine-guided instruction (Bowen et al. 2014;
Joyce et al. 2015; alperto, Couch, and Harmon 2016). Bettinger et al. (2017) ﬁnd that
negative eﬀects of online course-taking are particularly large for students with lower
prior grade point averages.

Descriptive ﬂipped classroom research ﬁnds mixed results.1 However, porque
of these studies’ nonrandom designs, diﬀerences between student outcomes in the
ﬂipped and standard classrooms could be due to diﬀerences in course material, en-
structor quality, student preparation or characteristics, or other factors. Our study is
most akin to Wozny, Balser, and Ives (2018), whose randomized controlled trial ﬁnds
that the ﬂipped classroom boosted students’ scores in econometrics. One drawback is
that the authors’ seven course sections serve as the sample for the study. debido a la
small number of sections, they randomize the ﬂipped or standard teaching methods
across lectures within the same sections, which could produce spillover eﬀects.2

1. See Lage, Platón, and Treglia (2000); Bergmann and Sams (2009); McLaughlin et al. (2014); Schultz et al. (2014);
Findlay-Thompson and Mombourquette (2014); Davies, Dean, and Ball (2013); Overmyer (2014); Swoboda and
Feiler (2016).

2. Por ejemplo, if students observe increased retention after ﬂipped classrooms, they might ask more follow-up
questions or focus more of their study eﬀorts on material from standard lectures to compensate. Wozny, Balser,

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

365

Effects of Flipped Classroom Instruction

Our study contributes to this relatively understudied topic by running a randomized
controlled trial with a large number of class sections, estudiantes, and instructors, and by
holding all aspects other than the ﬂipped classroom constant, including the course ma-
terials, course content, and instructors. All twenty-nine instructors of the two courses
participated in the experiment, allowing us to understand the eﬀects of ﬂipped class-
rooms for a range of instructor types, not just those most motivated to pedagogically
innovate. We provide the ﬁrst causal ﬂipped classroom evidence from classroom level
randomization and the ﬁrst ﬂipped classroom estimates for two separate subjects.

The next section provides background on West Point and the ﬂipped classroom.
Sección 3 describes the design of the experiment and section 4 details the characteristics
of the students, classrooms, and instructors in the study. Sección 5 lays out the empirical
estructura. Sección 6 presents the results, provides evidence to explain the diﬀerential
eﬀects in math and economics, and explores the equity implications. Sección 7 oﬀers
concluding thoughts.

I N S T I T U T I O N A L D E TA I L S

2 .
The United States Military Academy at West Point is a four-year undergraduate in-
stitution that prepares students to become military oﬃcers. Además, West Point
shares many characteristics with small, liberal arts schools. Each cohort has approxi-
mately 1,100 cadets and they complete a twenty-seven-course liberal arts curriculum.
West Point caps class sizes at eighteen cadets per instructor and the average class has
sixteen cadets per instructor. West Point also has some unique characteristics that dis-
tinguish it from other postsecondary institutions. Cadets must receive permission to
miss class, there is a high level of discipline in the classroom, the cadet population is
predominately male, y, considering the propensity for cadets to serve in combat after
graduation, cadets are probably less risk-averse than the average U.S. college student.3

Admissions
West Point has a competitive admissions process. Applicants must receive a nomina-
tion from one of their Congressional representatives and must demonstrate physical
ﬁtness. Como resultado, West Point students are more athletic and geographically diverse
than typical universities. Estados Unidos. News and World Report ranks West Point as num-
ber eighteen in their list of National Liberal Arts Colleges.4 West Point students have a
mean SAT score of 627 in Reading and 645 in Math (out of a possible score of 800 en
cada), comparable to similarly ranked liberal arts colleges (West Point 2019).

Faculty
West Point has three types of faculty members: senior military faculty, civilian profes-
sors, and junior rotating military faculty. The senior military faculty serve permanently
at West Point and most hold a doctorate degree. Civilian professors typically serve on

and Ives (2018) produce a variety of robustness checks that suggest the teaching methods of previous lessons
do not impact test scores from current lessons.

3. See table 1 of Carter, Greenberg, and Walker (2017) for a comparison of West Point to similarly ranked liberal

arts colleges and to all four-year postsecondary institutions.

4. Ver https://www.usnews.com/best-colleges/west-point-2893/overall-rankings.

366

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Mesa 1. Class Size and Teaching Load

Full Sample

Matemáticas

Ciencias económicas

Todo
(1)

2.8
(0.9)

1328

16.6
(1.0)

Tratamiento
(2)

Control
(3)

1.4
(0.5)

661

16.5
(1.0)

1.5
(0.5)

667

16.7
(0.9)

Todo
(4)

2.5
(0.8)

852

16.7
(0.6)

Tratamiento
(5)

Control
(6)

1.4
(0.5)

435

16.7
(0.6)

1.4
(0.5)

417

16.7
(0.7)

Todo
(7)

3.2
(1.0)

476

16.4
(1.4)

Tratamiento
(8)

Control
(9)

1.6
(0.5)

226

16.1
(1.5)

1.7
(0.5)

250

16.7
(1.2)

Number of instructors

Number of sections

Sections taught by instructor

Number of students

Average class size

Notas: This table describes the number of instructors, secciones, and students in the treatment and control groups. Standard deviations are
reported in parentheses.

the faculty for a prolonged period and have PhDs. The junior military faculty spend
three years teaching and hold a master’s degree in their area of instruction. The junior
military faculty teach lower level electives and the more basic core curriculum courses,
while civilian and senior military faculty teach more advanced courses.

Curriculum
The ﬁrst two years at West Point are almost exclusively core curriculum courses. En
order to accommodate the large enrollment, West Point oﬀers many sections of these
courses. Students are highly incentivized to do well because grades determine job place-
ment after graduation. Because of the importance of course performance, West Point
prioritizes standardizing courses and course grading. Course directors set the syllabus,
lesson objectives, assignments, and exams so that they are consistent across all instruc-
tors teaching the course. New instructors receive training from the course director to
further ensure standardization across class sections. The course standardization and
randomization of students to course sections makes West Point an ideal place to study
the ﬂipped classroom.

3 . E X P E R I M E N TA L D E S I G N
The experiment took place in the 2016 Fall semester in two required core courses: En-
troduction to Calculus and Principles of Economics. We selected these courses because
their quantitative nature lends themselves to interactive problem-solving. Este método
links well with the active teaching style the ﬂipped classroom strategy utilizes.

Cadets take the math class in their ﬁrst semester at West Point and take Princi-
ples, their ﬁrst economics course, in their sophomore year. Students with stronger
math backgrounds and students interested in majoring in Economics can take more
advanced and in-depth versions of these classes. Because of the small number of ad-
vanced classes, we excluded these class sections from the study.

We also chose these two courses because they have a large number of courses sec-
ciones, estudiantes, and faculty. All faculty who taught at least two sections of these courses
participated in the study. Twenty-nine faculty members, 80 class sections, y 1,328
students participated in the experiment (ver tabla 1). Math constituted a larger portion

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

367

Effects of Flipped Classroom Instruction

of the sample with 20 instructors and 852 estudiantes, mientras 9 instructors and 476 estu-
dents in economics participated. Forty class sections were randomly assigned to the
ﬂipped classroom treatment, 26 in math and 14 in economics. Otro 40 secciones
were randomly assigned to the standard classroom control (25 in math and 15 in eco-
nomics). Each instructor taught at least 1 ﬂipped and 1 control classroom. De término medio,
instructors taught 2.8 classes in the study. While random assignment happened at the
classroom level, the registrar randomly assigned students to class sections, balancing
on baseline academic ability.

We selected a three-lesson unit from both the math and economics courses to con-
duct the experiment. This discrete group of lessons enabled strict adherence to the
experimental design. We think our experiment gives a lower bound on the impact of
the ﬂipped classroom for several reasons. Primero, the faculty involved are all new to the
ﬂipped classroom format and we would expect their eﬀectiveness to improve with more
experiencia. Segundo, our experiment occurs in the middle of the courses: in the lessons
14 a través de 16 of the math course and in lessons 22 a través de 24 for economics (after the
microeconomics units and before the macroeconomics units). It may be challenging
for faculty to switch and be disruptive for students. Por último, we observe instructors in
both the standard lecture and the ﬂipped classroom, so instructors cannot focus their
preparation time on one type of class.

We chose the vector and personal ﬁnance5 units for the study because neither re-
quired nor built upon prior knowledge of the subject area. The math unit covered
dot products and parametric equations. The personal ﬁnance unit in economics cov-
ered budgeting, present discounted value, and retirement and investment calculations.
Some young people are exposed to the basics of personal ﬁnance through self-study,
interactions with parents, and previous employment. A diferencia de, students have little
to no prior knowledge of the vector math covered in the unit.

The course directors created and lectured in the videos. We chose to have one set
of videos for each course (instead of having each instructor create their own video se-
ries) to ensure that each treated section had access to the same quality video with an
experienced lecturer. We vetted the videos for consistent formatting between math and
economics and piloted them in the summer prior to the experiment. We posted the
videos to an internal West Point Web site that required students to log in to view the
videos. This allowed us to monitor each student’s video watching. The Web site allowed
any student with the link to watch the video as many times as they wanted.6

Class Structure
Consistency across course sections is highly valued at West Point because grades fac-
tor into students’ job placement after graduation. This standardization across sections
strengthens our design and allows us to hold all components of the course constant,
except for the “ﬂipped” aspects. Cifra 1 shows the timing of the diﬀerent course

5. The primary purpose of the ﬁnancial literacy unit is to prepare students for managing a large (upwards of
$30,000) loan they receive in the middle of their junior year. The loan gives students upfront capital to purchase
life necessities (including vehicles, uniforms, and furniture) to begin their Army career.

6. As a backup, the videos were also loaded to YouTube and the instructors were informed of this alternate capabil-
ity to help them troubleshoot viewing problems during execution of the experiment. We cannot track YouTube
video watching except for student reported surveys.

368

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Cifra 1. Class Format for Treatment and Control Groups

componentes. Both the ﬂipped and standard classrooms were assigned identical read-
ings to complete before lecture.

Además, students in the ﬂipped classroom were tasked with watching a 20-
minute video lesson before each of the three lessons in the unit. Students in the
treatment group were e-mailed instructions to watch videos before class and given re-
minders during lecture. They were informed that instructors would track whether they
watched the videos. The e-mail emphasized that watching the video was important to
their learning and they would lose participation points for not watching. Instructors
were directed not to inform students about their participation in the experiment.

Upon arriving to class, instructors took attendance and made class announcements
for both the treatment and control groups. Entonces, the treatment classrooms proceeded
with a question-and-answer session with the instructor about the video for that lesson.
The instructors were provided clear guidance to avoid lengthy lectures during this pe-
riod, but to use the opportunity to clarify speciﬁc questions.

Then the ﬂipped classrooms worked on a problem set with ten to ﬁfteen practice
problems that linked directly to the lesson objectives. The video covered the material in
the problem sets and modeled how to solve similar problems. Instructors were given
strict guidelines to not conduct a traditional lecture, but instead guide the students
through the worksheet and answer any questions they might have. Implementation
of the problem set varied from instructor to instructor. Some instructors would have
the cadets complete one problem at a time and then have the class discuss the solu-
ción. Other instructors would let the cadets work the entire way through the worksheet
uninterrupted and circulated the room to answer individual questions.

The control group experienced little change to the normal class procedures. Después
class administrative tasks, the instructors delivered a standardized lecture to the cadets
that mirrored the content in the videos. The math course director provided a detailed
script for instructors to follow in delivering the lesson. The economics control group
class delivered the lecture following the same slides built into the economics ﬂipped
classroom video. During the course of the lecture, both math and economics instructors
worked through quantitative problems on the board. Instructors could take questions
throughout the lecture.

As the control group students departed at the end of the class period, instructors
handed out practice problem worksheets that the ﬂipped classroom students worked
on during class. Instructors encouraged students to complete the worksheets prior to
the next class but instructors did not collect or grade the worksheets. Por lo tanto, el

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

369

Effects of Flipped Classroom Instruction

control group students had the same opportunity to practice problems as the ﬂipped
aula. Course directors circulated the classrooms during the experiment to check
for proper implementation of both the ﬂipped and control classrooms.

There were no strong incentives to watch the video or to complete the practice prob-
lems outside of class. Because of the military nature of West Point, students may be
more likely to follow instructions than the typical college student. Sin embargo, survey data
ﬁnd (see descriptive statistics below) that students spend less time preparing for class
than recommended and do not all watch the videos in full. If compliance for class in-
structions is stronger among West Point students relative to other college settings, él
aﬀects both the treatment and control students similarly, since they are each instructed
to do work outside of the classroom (p.ej., watching videos or completing practice prob-
lemas). West Point has a strict attendance policy, so we are able to control the amount
of time that students spend in class with instructors. If the ﬂipped classroom changes
student attendance in less strict settings, then we will not see the eﬀects of this in our
estudiar.

Outcomes
After the completion of the three-lesson block, students in both the ﬂipped and stan-
dard classrooms took an in-class quiz that covered the material of the experiment’s
lecciones. The in-class quiz accounted for 3 percent of the math course and 3.5 por ciento de
the economics course grades. Both the treatment and control groups received identical
quizzes.

To test for fade-out of knowledge or whether students improved their knowledge of
the material before the ﬁnal exam, we analyze performance on the unit-speciﬁc ques-
tions on the ﬁnal exams and the overall ﬁnal exam grade. For math, the exam was
administered in May 2017 and for economics the exam took place in December 2016.
For both classes, the exam accounted for 25 percent of the course grade.

4 . D E S C R I P T I V E S TAT I S T I C S A N D I M P L E M E N TAT I O N D E TA I L S
Student Characteristics and Covariate Balance
The random assignment of students and instructors eﬀectively balanced the demo-
graphic composition and baseline academic ability of the standard and ﬂipped class-
rooms. columnas 1 y 2 de mesa 2 show that students assigned to the control group, con
classes taught in a traditional lecture format, and students assigned to the treatment
grupo, with classes taught in a ﬂipped classroom format, have similar average baseline
características. Women constitute about 21 percent of students in both groups. Apenas
60 percent of students in the experiment are white, African American students account
for approximately 17 percent of the sample, and Hispanic students account for just over
10 por ciento. Nearly one in six students has prior military service as enlisted soldiers and
the average composite ACT score for both groups exceeds 28.7 The ﬂipped and stan-
dard classroom students also scored similarly on West Point’s College Entrance Exam
Rango (CEER), an admissions tool that factors in high school class ranking, SAT or ACT
puntuaciones, physical ﬁtness, West Point faculty evaluations, and extracurricular activities.

7. For students who did not take the ACT, we map SAT scores to comparable ACT scores. The highest possible

ACT score is 36, con 21 being the average score.

370

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Mesa 2. Student Characteristics and Covariate Balance

Significar

Difference between Treatment and Control

Tratamiento
(2)

Full Sample
(3)

Math Classes
(4)

Economics Classes
(5)

Baseline Characteristic

Female

Blanco

Negro

Hispano

Age

Prior military service

Composite ACT

College Entrance Exam Rank score

Control
(1)

0.205
(0.404)

0.600
(0.490)

0.175
(0.381)

0.111
(0.314)

19.461
(1.155)

0.165
(0.371)

28.085
(3.448)

6.017
(0.682)

0.212
(0.409)

0.611
(0.488)

0.169
(0.375)

0.100
(0.300)

19.390
(1.231)

0.159
(0.366)

28.324
(3.350)

6.053
(0.645)

p-value (joint χ 2 prueba)
Observaciones

667

661

−0,002
(0.014)

0.002
(0.024)
−0,006
(0.022)
−0,003
(0.012)
−0.014
(0.044)

0.006
(0.016)

0.127
(0.141)

0.017
(0.030)

0.966

1,328

−0,002
(0.019)
−0,008
(0.036)

0.000
(0.028)
−0,007
(0.017)
−0.043
(0.055)

0.005
(0.020)

0.150
(0.187)

0.031
(0.044)

0.890

852

−0,003
(0.019)

0.018
(0.025)
−0,016
(0.034)

0.002
(0.016)

0.033
(0.074)

0.008
(0.026)

0.091
(0.214)
−0,004
(0.034)

0.951

476

Notas: This table reports descriptive statistics of students in the experiment. Columna 1 reports mean characteristics of
the control group (students in classrooms with the standard lecture format) and column 2 reports means for students in
the treatment group (flipped classrooms). Standard deviations are reported in parentheses. columnas 3, 4, y 5 informe
coefficient estimates from a regression of the baseline characteristics on an indicator variable that equals one if a student
is assigned to a flipped classroom. The regressions used to construct estimates in columns 3, 4, y 5 include (curso) X
(instructor) y (curso) X (hour) fixed effects. Errores estándar, clustered on classrooms (each instructor-hour combination),
se informan entre paréntesis. The reported p-values come from a joint test of the null hypothesis that all coefficients are
equal to zero.

Columna 3 de mesa 2 reports the regression-adjusted diﬀerences between students as-
signed to treatment classrooms and students assigned to control group classrooms. Nosotros
construct these diﬀerences from regressions that include instructor ﬁxed eﬀects and
class schedule block ﬁxed eﬀects. The diﬀerences between treatment and control class-
room characteristics are all small and statistically insigniﬁcant, suggesting assignment
to treatment or control groups was as good as random. A test of the joint-hypothesis
that all diﬀerences in baseline characteristics equal 0 yields a p-value of 0.966 (abajo
row of table 2), further suggesting that the randomization was eﬀective. These similar-
ities would indicate that any diﬀerence in the performance of the ﬂipped and standard
classrooms can be attributed to the treatment of the ﬂipped classroom.

columnas 4 y 5 de mesa 2 report the same covariate balance checks after restricting
the sample to math and economics classes, respectivamente. As with the estimates reported
in column 3, there are no noticeable diﬀerences between the treatment and control
groups for either class type.

Cuadro A.1, which is available in a separate online appendix that can be accessed
on Education Finance and Policy’s Web site at https://doi.org/10.1162/edfp_a_00314, ex-
plores attrition rates for taking the unit quiz and ﬁnal exam, the key outcomes of in-
terest. Approximately 97 percent of the sample takes the quiz and the treatment and
control groups have similar quiz-taking rates. The ﬁnal exam attrition rate is twice as
large as the quiz attrition. While attrition is not statistically signiﬁcantly diﬀerent in the

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

371

Effects of Flipped Classroom Instruction

economics treatment and control groups, the treatment group in math has a marginally
signiﬁcantly higher attrition rate relative to the control group. We estimate Lee (2009)
treatment-eﬀect bounds and ﬁnd that attrition does not bias our ﬁndings.

Classroom Characteristics
Instructors and Class Size
Consistent with West Point’s small class sizes, the average class in the study has 16.6
estudiantes (ver tabla 1). Flipped and standard math classes both had student–teacher ra-
tios of 16.7. Economics classrooms had similar class sizes of 16.1 for ﬂipped and 16.7 para
standard lectures.

A total of twenty-nine instructors participated in the experiment, with twenty from
math and nine from economics. The majority of instructors were military oﬃcers with
three or fewer years of teaching experience. The rest of the instructors were senior mil-
itary oﬃcers (three in math, one in economics) and civilian faculty (also three in math,
one in economics). These faculty each had at least ﬁve years of teaching experience.8

Before teaching any classes, all U.S. Military Academy instructors must pass a rig-
orous six-week training course where they learn best teaching practices, observe ex-
perienced instructors teach summer classes, and present practice lectures to a panel
of senior military and civilian faculty members. The experiment took place at roughly
the mid-point of the fall semester, so all instructors had a minimum of two months of
teaching experience, plus the six-week training course, before the experiment began.

Class Time Allocation
We administered an instructor survey at the conclusion of the course to gather descrip-
tive information about how the ﬂipped and standard classrooms functioned in practice.
Encima 86 percent of instructors completed the survey. Table A.1 in the online appendix
shows no diﬀerential attrition across whether treatment or control students had an in-
structor who responded to the survey. We also administered student surveys at the end
of the class and collected video-watching data that we discuss below.

Mesa 3 displays the results of the survey and conﬁrms that faculty carried out the
ﬂipped classroom and standard classroom models as instructed. Consistent with class-
room observations, faculty reported using the class time as instructed: on average,
instructors reported spending 85 percent of the class time lecturing in the standard
aula, relative to 8 percent of the time in the ﬂipped classroom.

The ﬂipped classroom also involved more in-class group and independent work
than the standard lecture: Instructors said that students worked in groups 76 por ciento
of the time in ﬂipped classrooms and 5 percent of the time in standard classrooms. Matemáticas
instructors reported that students worked in groups for only 1.5 percent of class time

8. After randomizing classrooms to treatment and control groups, but before the start of the school year, el
Math Department added an additional instructor. The Math Department assigned the new instructor to teach
three sections that had been assigned to three separate math instructors in our experiment. These sections
included two treatment and one control. We conﬁrmed with the Math Department that our experiment had no
bearing on the decision to bring in an additional instructor, including the decision about which sections the
new instructor would teach. As a result of this swap, the three instructors who lost a section taught only one
section and therefore had no variation in treatment and control sections during the experiment. We did not
adjust the random assignment after this section swap occurred and we did not permit instructors to self-select
their sections into treatment or control groups.

372

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Mesa 3. Classroom Characteristics

Full Sample

Matemáticas

Ciencias económicas

Flipped
(1)

Estándar
(2)

Flipped
(3)

Estándar
(4)

Flipped
(5)

Estándar
(6)

Grupo A: Class Time Allocation (in Percent)

Lecturing

Students working in group

Students working alone

Answering individual questions

Answering questions for class

Review old material

Other/administration

8.0
(11.9)

76.0
(30.2)

30.0
(38.9)

42.0
(26.7)

36.0
(22.9)

5.0
(10.2)

10.0

(12.5)

85.0
(16.1)

5.0
(10.2)

6.0
(13.1)

53.0
(34.9)

6.0
(10.9)

9.0

8.8
(12.3)

73.5
(32.4)

36.8
(42.5)

44.1
(30.0)

44.1
(18.8)

7.4
(11.7)

8.8

88.2
(15.6)

1.5
(6.1)

4.4
(13.2)

5.9
(14.1)

50.0
(33.1)

8.8
(12.3)

7.4

6.3
(11.6)

81.3
(25.9)

15.6
(26.5)

37.5
(18.9)

18.8
(22.2)

0.0
(0.0)

12.5

78.1
(16.0)

12.5
(13.4)

9.4
(12.9)

6.3
(11.6)

59.4
(39.9)

0.0
(0.0)

12.5

(12.2)

(12.3)

(11.7)

(13.4)

Grupo B: Percent of Student Who Do the Following at Least Once during Class Time

Work in group

Work alone

Asking questions

Answering questions

Taking notes

Paying attention

norte

76.0
(25.5)

23.0
(27.9)

56.0
(19.5)

48.0
(24.9)

47.0
(28.2)

74.0
(18.4)

25.0

11.0
(24.0)

18.0
(30.2)

45.0
(20.4)

45.0
(19.1)

75.0
(21.7)

65.0
(17.7)

25.0

77.9
(24.8)

25.0
(28.0)

57.4
(21.2)

44.1
(24.3)

35.3
(21.8)

77.9
(15.0)

17.0

11.8
(26.7)

41.2
(19.6)

39.7
(19.9)

73.5
(22.5)

61.8
(17.9)

17.0

71.9
(28.1)

18.8
(29.1)

53.1
(16.0)

56.3
(25.9)

71.9
(24.8)

65.6
(22.9)

8.0

9.4
(18.6)

31.3
(34.7)

53.1
(20.9)

56.3
(11.6)

78.1
(20.9)

71.9
(16.0)

8.0

Notas: This table describes the flipped and standard classrooms using data from a post-study instructor survey.
Panel A displays the average percent of time instructors reported spending on different activities. Items did
not need to add up to 100 percent because some activities could happen simultaneously (p.ej., answering
individual questions and having students work in a group). Panel B shows the percent of time the instructor
observed students doing an activity in class. Standard deviations are reported in parentheses.

in the standard lecture, while group work was more common in the economics stan-
dard lectures: constituting 12.5 percent of time. Students also worked alone on practice
problems more frequently in the ﬂipped relative to the standard classrooms, con el
starkest diﬀerence in math (36.8 percent versus 4.4 por ciento) and a smaller diﬀerence
in economics (15.6 percent versus 9.4 por ciento).

Math faculty spent more time answering questions for the whole class in the stan-
dard classroom than the ﬂipped classrooms, but more time circulating around the class-
room to answer individual student questions in the ﬂipped classroom. Economics in-
structors reported a similar pattern but with a larger diﬀerence between the amount of
time answering questions in front of the class.

Instructors spent small and similar amounts of time reviewing old material and

other tasks in the ﬂipped and standard lectures.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

373

Effects of Flipped Classroom Instruction

Student Behaviors
We also asked instructors to report behaviors of students during class time (see panel B
in table 3). Some student behaviors reﬂect clear diﬀerences between ﬂipped and stan-
dard classrooms: Faculty reported 76 percent of students working in groups in a typical
ﬂipped classroom, comparado con 11 percent of students in the standard lecture.

Other survey responses suggest diﬀerent implementation in the math sections
compared to the economics sections. In math, working alone was more than twice as
common in the ﬂipped versus the standard classroom. Sin embargo, in economics, the re-
lationship was ﬂipped. Además, while asking and answering questions were more
common in the math ﬂipped classrooms relative to the standard lectures (consistent
with the ﬂipped classroom model), students asked and answered questions at sim-
ilar rates in the ﬂipped and standard economics classrooms. También, math instructors
reported higher note-taking rates in the standard lecture than the ﬂipped classroom,
while economics instructors reported only a slightly higher rate in the standard class-
rooms. Juntos, these survey results suggest that the math classes implemented the
ﬂipped classroom model more fully than economics.

Por último, instructors perceived that more students paid attention in math in the ﬂipped
aula (77.9 percent versus 61.8 por ciento), while attention in economics was greater
in the standard lecture (71.9 percent versus 65.6 percent in the ﬂipped classroom).9
This shows that math instructors found the ﬂipped classroom to be more engaging,
while economics faculty found the standard lecture preferable for students.

Time and Activities Outside of Class
Video Watching
Using the unique logins to the West Point Web site that hosted the videos, we tracked
student streaming of the class videos. Panel A of table 4 shows that almost 80 por ciento de
math and 73 percent of economics students in the ﬂipped classrooms watched at least
some of a video. De término medio, students watched roughly two out of three of the videos.
As expected, the control group rarely accessed the video: Solo 2 percent of math and 3
percent of economics standard lecture students ever logged in. To log in, they needed
to get the link from someone in a ﬂipped classroom.

We track the proportion of the video data that streamed to students’ computers and
ﬁnd that on average students watch roughly 50 percent of the video content in math
and economics.10 In a survey at the end of the course, we asked students about how they
watched the videos. The responses in panel B of table 4 show that over three fourths
of the math ﬂipped classroom students report repeating sections when they watch the
video compared with only 37 percent of economics students. Students reported low
rates of multitasking while watching the videos in math (6 por ciento), but higher rates
(31 por ciento) in economics.

9. The columns in table 3, panel A, do not need to add up to 100 percent because multiple activities can occur at
al mismo tiempo. Por ejemplo, instructors can answer individual questions while students work in groups.
10. We measure percent of video watched by dividing the number of bytes downloaded by the total number of bytes
for each lesson. If students streamed a lesson’s video more than once, we take the session where they watched
the largest proportion of the video. This conservatively measures student video watching by undercounting the
proportion of the video watched if students do not restart from the beginning.

374

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Mesa 4. Video Watching

Matemáticas

Ciencias económicas

Flipped
(1)

Estándar
(2)

Flipped
(3)

Estándar
(4)

Grupo A: Video Watching

Ever watched video

Proportion of videos students watched

Proportion of video length watched

norte

0.798
(0.402)

0.559
(0.370)

0.453
(0.304)

435

0.022
(0.145)

0.008
(0.056)

0.006
(0.050)

417

Grupo B: Video Watching Style

Repeated sections

Multitasked while watching

norte

0.770
(0.421)

0.066
(0.249)

318

—
—

—

0.730
(0.445)

0.544
(0.398)

0.543
(0.399)

226

0.371
(0.485)

0.308
(0.463)

143

0.036
(0.187)

0.015
(0.080)

250

—
—

—

Notas: This table reports average video watching behaviors of students in flipped and stan-
dard classrooms. Panel A data come from log-in and streaming data to the Web site that
hosted the video lectures. Panel B data come from a post-study student survey. Estándar
deviations are reported in parentheses.

Class Preparation
The random assignment of classrooms to ﬂipped or standard lecture format ensures
that we estimate the causal impact of the ﬂipped classroom. Sin embargo, the ﬂipped class-
room format could increase the time spent on the class or time spent on practice prob-
lemas. If these changes inﬂuence student outcomes, then it is possible the eﬀects are
driven by changes in the amount of time students spend on the class and not from the
ﬂipped classroom format.

We ﬁnd no evidence that students in the treatment group spend more time prepar-
ing for the class than students in the control group. Panel A of table 5 shows that stu-
dents in the math standard lecture spend approximately 12 minutes more preparing for
lecture compared with the math ﬂipped classroom students (1.35 hours compared with
1.15 horas). This 12-minute diﬀerence is statistically signiﬁcant, though a relatively small
amount of time. Because the ﬂipped classroom students spend slightly less time prepar-
ing for lecture, the treatment eﬀects cannot be driven by students spending more time
preparing. Economics students report spending less time preparing for lecture overall
(0.87 y 0.73 hours for ﬂipped and standard lectures, respectivamente).11 This suggests
that students spend similar amounts of time preparing for lecture by watching videos
or doing the assigned readings.

11. While self-reported data are not ideal, students knew their responses would not aﬀect their grade or be viewed
by their instructors. The average student responded that they spent 30 a 50 percent less time preparing for
class than West Point suggests (2 hours per class, see panel A of table 5). This signals students’ willingness to
give non-favorable answers. Además, the ﬂipped and standard classrooms did not have diﬀerent incentives
to over- or underreport their course preparation, so any measurement error should be consistent across the two
grupos.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

375

Effects of Flipped Classroom Instruction

Mesa 5. Student Preparation Outside of Class

Matemáticas

Ciencias económicas

Significar

P-value of Flipped
= Standard

Significar

P-value of Flipped
= Standard

Flipped
(1)

Estándar
(2)

(3)

Flipped
(4)

Estándar
(5)

Grupo A: Time Spent on Course

Hours spent preparing for lecture
(watching videos or reading
combined)

norte

Hours spent on practice problems

outside of class

norte

Complete all of the assigned
reading for every class

Complete part or all of the

assigned reading for every class

Never read for class

norte

1.148
(0.485)

318

0.910
(0.545)

318

0.233
(0.423)

0.830
(0.376)

0.082
(0.274)

318

0.000

0.980

1.346
(0.481)

325

0.909
(0.631)

325

Grupo B: Course Reading

0.563

0.381

0.016

0.252
(0.435)

0.855
(0.352)

0.037
(0.189)

325

0.867
(0.384)

143

0.661
(0.516)

143

0.105
(0.307)

0.664
(0.474)

0.119
(0.325)

143

0.729
(0.476)

144

0.865
(0.598)

144

0.125
(0.332)

0.590
(0.493)

0.229
(0.422)

144

Grupo C: Rating for Helpfulness of Learning Tool (0 = Not helpful, 1 = Very helpful)

Reading (for those who read)

norte

Video (just treatment)

norte

Reading (just treatment)

norte

Readings (just control)

norte

0.410
(0.230)

296

0.615
(0.262)

301

0.410
(0.230)

296

0.316

0.429
(0.229)

304

0.225
(0.325)

417

0.388
(0.233)

123

0.353
(0.231)

129

0.388
(0.233)

123

0.350
(0.193)

115

0.036
(0.126)

250

(6)

0.007

0.002

0.595

0.196

0.014

0.171

Notas: This table reports end-of-course survey responses from students in flipped and standard classrooms. columnas 1, 2, 4, y 5 show the
mean responses and columns 3 y 6 show the p-value of the test of whether the flipped and standard classroom means are equal. Estándar
deviations are reported in parentheses.

Math ﬂipped classroom and standard classroom students report spending similar
amounts of time outside of class on practice problems (see panel A of table 5). Porque
the treatment group worked on the problems in class, this signals they have overall
more time to work on practice problems relative to the control group, which could
contribute to the higher average scores on the quiz. A diferencia de, economics students
in standard classrooms spend signiﬁcantly more time on practice problems outside of
class relative to their ﬂipped classroom peers.

Flipped classroom students are also similarly likely to complete some or all the read-
ings for class (see panel B of table 5). Approximately 24 percent of math students report
completing the readings for every class and over 83 percent complete some of the read-
ings for each class. Reading completion is less common in economics: Encima 10 por ciento

376

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

of students report completing all of the readings and over 59 percent complete some
of the readings.

5 . E M P I R I C A L F R A M E WO R K
We estimate the equation below to compare outcomes between students in the ﬂipped
(treatment) classrooms to students in the standard lecture (control) classrooms:

Yi jh = α + βTjh + κ j + λh + γ (cid:2)

Xi + (cid:8)i jh,

(1)

where Yi jh is the exam score of student i with instructor j during class-hour (schedule
block) h. Tjh is a binary variable that equals 1 for students in the ﬂipped classrooms
y 0 for students in the standard lecture classrooms. Xi is a vector of individual con-
trols, including race, género, edad, prior military service, and composite ACT and CEER
puntuaciones. Ecuación 1 also includes instructor ﬁxed eﬀects, κ j, to control for instructor abil-
ity that is constant across the type of class taught. Class-hour ﬁxed eﬀects, λh, captura
unobserved mean diﬀerences in academic performance across class hours. Due to the
random assignment of ﬂipped-classroom status, estimates of β capture the causal ef-
fect of learning in a ﬂipped classroom environment. We cluster standard errors at the
classroom level.

6 . RESULTADOS
We ﬁnd the ﬂipped classroom has a short-term positive eﬀect that fades by the time
students take the ﬁnal exam. Mesa 6 reports the estimates of equation 1 for the quiz at
the end of the unit, the questions speciﬁc to that unit in the ﬁnal exam, and the overall
ﬁnal exam score. All test scores are standardized to have a mean of 0 and standard
desviación de 1. In our study, ﬂipped classrooms increase quiz scores by 0.165 standard
deviations relative to standard lecture classrooms (columna 1 de mesa 6). The results
remain similar after controlling for student-level baseline covariates in column 2.12

Subsequent columns of table 6 reveal that only math classes experience the short-
term positive eﬀect of ﬂipped classrooms. columnas 3 y 4 indicate that the ﬂipped
classroom environment improved test scores of students in math classes by roughly 0.3
desviaciones estandar. We see positive eﬀects for both rote questions that involve memo-
rization and more advanced questions that require problem-solving.

In contrast to the math results, the average test scores among economics students
in ﬂipped classrooms were roughly 0.07 standard deviations lower than the average
test scores of economics students in standard classrooms, though this estimate is only
marginally signiﬁcant and indistinguishable from zero when we correct for the number
of clusters.13,14

12. Results with and without instructor ﬁxed eﬀects are also similar.
13. With only twenty-nine economics sections, our clustered standard errors for economics classes are potentially
biased downwards. To investigate this further, table A.2 in the online appendix reports conventional standard
errores, robust standard errors, and standard errors constructed from section-level unit-quiz means. The results
of this investigation suggest that the marginally signiﬁcant negative estimate among economics classrooms
is indistinguishable from zero when we correct for the small number of clusters (columna 4). Our positive
estimates for math classrooms, sin embargo, are statistically signiﬁcant regardless of how we estimate standard
errores.

14. Data were not available to analyze rote versus problem-solving eﬀects for economics.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

377

Effects of Flipped Classroom Instruction

Mesa 6. Effect of Treatment on Academic Outcomes

Full Sample

Matemáticas

Ciencias económicas

(1)

(2)

(3)

(4)

(5)

(6)

0.165***
(0.048)

0.148***
(0.044)

0.304***
(0.062)

0.282***
(0.058)

1,281

0.035
(0.057)

1,254
−0.020
(0.042)

1,262

1,281

0.025
(0.053)

1,254
−0,035
(0.033)

1,262

Unit quiz

norte

Final: Unit-specific questions

norte

Final: Overall score

norte

Rote questions on quiz

norte

Problem-solving questions on quiz

norte

Instructor and time block controls

Demographic controls

Clusters (aula)

818

0.057
(0.071)

801

0.027
(0.060)

801
0.250***
(0.058)

848
0.223***
(0.076)

848

818

0.039
(0.061)

801

0.001
(0.048)

801
0.238***
(0.060)

848
0.205***
(0.068)

848

−0.055
(0.043)

463

0.000
(0.094)

453
−0.093*
(0.050)

−0.071*
(0.037)

463
−0,004
(0.096)

453
−0.100**
(0.039)

461

Notas: This table reports estimates from regressions of exam scores on an indicator for being assigned to a flipped classroom.
All scores are standardized to have a mean of 0 y una desviación estándar de 1 for each subject. Baseline controls include
instructor fixed effects and class hour (es decir., time block) fixed effects. Demographic controls include indicators for female, white,
negro, Hispano, and for having prior military service, plus linear terms for age, ACT score, and West Point’s College Entrance
Exam Rank score. Errores estándar, clustered on classroom, se informan entre paréntesis.
***Significativo en el 1% nivel; **significativo en el 5% nivel; *significativo en el 10% nivel.

To investigate whether the positive impact of ﬂipped classrooms on math classes
persists, we test the eﬀect of the ﬂipped classroom on the ﬁnal exam. We estimate the
eﬀect for both the overall score and for the questions speciﬁc to the experimental unit.15
We ensured that the diﬃculty and content of the ﬁnal exam questions for the experi-
mental unit material was comparable to that of the experimental unit quiz. Students in
the math-ﬂipped classrooms perform similarly to those in the standard lecture for both
the questions on the ﬁnal—speciﬁc to the experimental unit—and the ﬁnal exam over-
todo. The point estimates are positive but small, que van desde 0.039 a 0.057 standard
deviations, and are indistinguishable from zero. Although we cannot rule out positive
eﬀects on the order of one ﬁfth of a standard deviation, these estimates do suggest the
positive impact of the ﬂipped-classroom environment on math comprehension likely
faded with time. The overall ﬁnal exam scores between students in ﬂipped classrooms
and students in standard lecture classrooms did not vary, which is unsurprising because
the ﬂipped classroom experiment did not extend beyond the speciﬁc set of lectures de-
scribed above.

In online table A.3, we investigate the faded-out eﬀects by reporting the mean test
scores of treatment and control students for the quiz, and the questions speciﬁc to
the experimental unit on the ﬁnal. We ﬁnd that ﬂipped classroom students increase
their knowledge of the experimental unit’s content following the quiz: They score

15. The units that followed the experimental unit did not build upon knowledge from the experimental unit. En-

structors did not spend time on the experimental unit’s material after the quiz.

378

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

22 percent higher on the unit-speciﬁc ﬁnal exam questions relative to the quiz (see col-
umn 1 of table A.3). Because the quiz and ﬁnal questions cover the same content with
highly comparable questions, this denotes that average student knowledge of the sub-
ject grew over time. The control group also increased their average score from the quiz
to the ﬁnal and caught up to the ﬂipped classroom students: Both groups scored sim-
ilarly, on average, on the ﬁnal questions speciﬁc to the experimental unit. This means
the null eﬀects on the ﬁnal exam stem from the control group’s catching up and not
that the ﬂipped classroom students’ knowledge faded.

Mesa 6 reports no long-term test eﬀects for students in economics classes for the
unit-speciﬁc ﬁnal exam questions, suggesting that if the ﬂipped classroom had any ini-
tially deleterious eﬀects on economics students, they likely faded over time. Puzzlingly,
students in the ﬂipped classroom scored about 0.1 standard deviations lower on the ﬁ-
nal exam overall relative to students in the control group. Sin embargo, after we account
for the small number of clusters among economics students, this eﬀect is not statisti-
cally signiﬁcant (see the p-values in online table A.4). Como resultado, we think the small
diﬀerence in point estimates is due to noise and the small number of clusters.

Differential Effects Across Subjects
What explains the short-term positive eﬀects of the ﬂipped classroom in math and the
null eﬀects in economics? We chose these subjects because both are quantitative and in-
volve problem-solving, common features of ﬂipped classroom subjects. Sin embargo, mientras
both courses have problem-solving aspects, the math lessons included a higher propor-
tion of problem-solving questions than economics, which included relatively more rote
memorization. Because of this, perhaps the material in the math lecture lent itself bet-
ter to the ﬂipped classroom.

Además, prior to the study, the math department employed a more interactive
teaching style relative to economics. The math faculty could have been more conﬁdent
in implementing ﬂipped classrooms due to the teaching styles of the department. Ad-
ditionally, instructor preferences could impact their eﬀort and teaching eﬀectiveness.
Cifra 2 supports this hypothesis: It plots individual instructor eﬀects on student quiz
scores by whether instructors prefer ﬂipped or standard lectures. Math instructors who
preferred ﬂipped classrooms generate similar or larger learning gains for their ﬂipped
classrooms relative to their standard lectures. A diferencia de, math instructors who prefer
standard classrooms are either equally eﬀective in ﬂipped or standard classrooms, o
more eﬀective in standard lectures. Having a math instructor who prefers teaching the
ﬂipped classroom is linked to an approximate doubling of the ﬂipped classroom treat-
ment eﬀect, although the point estimate is noisy (ver tabla 7). All economics faculty
prefer standard lectures and all but one instructor has a stronger impact on their stan-
dard lecture students relative to their ﬂipped classroom students. This suggests that
instructor preferences could play a role in the impact of the ﬂipped classroom.

There are two issues with these survey data. Because we surveyed instructors at the
conclusion of the experiment, their preferences could have been inﬂuenced by how
eﬀective they thought they were in the ﬂipped versus standard classroom. Segundo, él
is possible that instructors exhibit the Hawthorne eﬀect: Instructors who do not like
the ﬂipped classroom may choose to reduce their eﬀort, which could then explain the

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

379

Effects of Flipped Classroom Instruction

Notas: This figure plots the instructor-specific quiz effects by whether they preferred the flipped or standard classroom in a post-study
survey. Estimates come from regressions of exam scores on an indicator for being assigned to a flipped classroom that include
baseline demographic controls and class hour fixed effects. Larger circles reflect more precise estimates: circle size is weighted by
the inverse variance of the effects.

Cifra 2.

Instructor Treatment Effects by Subject and Preference

null or negative outcomes for instructors who prefer the standard lecture. Juntos,
these complications suggest the need for instructors to actively want to implement the
ﬂipped classroom, along with training and support instructors.

We ﬁnd evidence of lower levels of engagement among economics ﬂipped class-
room students, both in and out of the classroom. This could contribute to the diﬀeren-
tial eﬀects between the subjects. Economics students rated the video less useful relative
to math students in our end of semester survey (see panel C of table 5). De término medio,
the math treatment group rated the video 50 percent more helpful than the reading.
A diferencia de, economics treatment group students rated the videos and readings simi-
larly useful. También, the economics students found the videos less helpful than the math
estudiantes: Economics students rated the videos an average of 0.35 out of 1 (con 0 denot-
ing not helpful and 1 denoting very helpful), compared with the math students’ rating
de 0.62 out of 1. Economics ﬂipped classroom students also report lower rates than
math students of repeating sections of the video and higher rates of multitasking while
watching the video. These ﬁndings suggest higher levels of engagement and interest
in the videos (a key component of the ﬂipped classroom model) among math students
compared with economics students.

Instructor survey responses also reveal higher levels of student engagement for the
ﬂipped classroom relative to the standard classroom for math. Mesa 3 shows that math
instructors observe higher rates of students paying attention, asking questions, trabajar-
ing in groups, and working independently in their ﬂipped classrooms relative to their
standard lectures. A diferencia de, economics faculty report lower rates of paying attention

380

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Mesa 7. Effect of Math Instructor Preferences on Unit Quiz Score

In flipped classroom

Instructor prefers teaching flipped classroom

Instructor prefers teaching flipped classroom × In flipped classroom

Instructor and time block controls

Demographic controls
R2

norte (Number of Students)

Clusters (aula)

(1)

(2)

(3)

0.304***
(0.062)
−0,006
(0.182)

0.083

785

0.151
(0.097)
−0.112
(0.228)
0.198*
(0.116)

0.152*
(0.092)
−0.135
(0.154)

0.168
(0.108)

0.085

0.206

785

Notas: This table reports estimates from a regression of unit-quiz exam scores on an indicator for being assigned
to a flipped classroom for students in the math section. columnas 2 y 3 report estimates from regressions that
interact the flipped classroom indicator with an indicator for whether the instructor preferred teaching a flipped
classroom in a post-study instructor survey. We do not report analogous estimates for economics classrooms
because all economics instructors preferred the standard lecture format over the flipped classroom format. Todo
scores have been standardized to have a mean of 0 y una desviación estándar de 1 and the controls are the same
as those described in table 5. Errores estándar, clustered on classroom, se informan entre paréntesis.
***Significativo en el 1% nivel; *significativo en el 10% nivel.

and similar rates of asking questions in the ﬂipped classrooms relative to their standard
classrooms—signaling lower levels of engagement. Por último, math instructors report in-
creased student–teacher and student–peer interaction in the ﬂipped classroom through
more question-asking and group work (ver tabla 3). A diferencia de, economics instructors
only report increased group work in their ﬂipped classrooms but similar amounts of
student questions. En resumen, the math ﬂipped classrooms had more student en-
gagement and student-teacher interactions than the math standard lectures, but eco-
nomics treatment and control classrooms had fewer diﬀerences for these classroom
características.

Por último, students in economics standard classrooms spend signiﬁcantly more time
on practice problems outside of class relative to the ﬂipped classroom economics stu-
abolladuras. A diferencia de, math students report spending similar amounts of time on practice
problems outside of class in the ﬂipped and standard classrooms. This additional prac-
tice time for the control group in economics may play a role in the similar test scores
of the treated and control groups in economics.

Equity Implications: Subgroup Effects
Subgroup analysis suggests the short-term positive impact of ﬂipped classrooms on
math performance is driven by men, white students, and higher-achieving students.
This can be seen in column 4 de mesa 8, which reports the ﬂipped classroom eﬀects
on quiz scores by student characteristics. The ﬂipped classroom has a large positive
eﬀect for men’s quiz scores (0.32 standard deviations gains) and a smaller and statis-
tically insigniﬁcant eﬀect for women. White students experience gains of 0.385 estan-
dard deviations higher on short-term learning, while black and Hispanic students have
statistically insigniﬁcant eﬀects close to zero. The diﬀerence in treatment eﬀects be-
tween white students and black or Hispanic students is statistically signiﬁcant (p-value

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

381

Effects of Flipped Classroom Instruction

Mesa 8. Subgroup Analysis

Female

norte

Male

p-value of female = male
Blanco

norte

Black or Hispanic

norte
p-value of white = non-white
ACT bottom quartile

norte

ACT 2nd quartile

norte

ACT 3rd quartile

norte

ACT top quartile

norte
p-value of ACT bottom quartile = Not bottom quartile
Instructor and time block controls

Demographic controls

Full Sample

Matemáticas

Ciencias económicas

(1)

(2)

(3)

(4)

(5)

(6)

−0,015
(0.103)

269
0.207***
(0.058)

0.025
(0.094)

269
0.175***
(0.052)

0.100
(0.131)

173
0.358***
(0.079)

0.084
(0.112)

−0.191
(0.156)

173

96
0.320*** −0,035
(0.043)
(0.072)

1012

645

367

0.077
0.223***
(0.066)

0.167
0.208***
(0.060)

0.148
0.405***
(0.088)

0.089
0.331
0.385*** −0.047
(0.071)
(0.077)

784

0.054
(0.077)

346

0.070

0.017
(0.075)

274
0.259**
(0.112)

251
0.207***
(0.055)

405

0.123
(0.087)

351

0.075

784

0.034
(0.079)

346

0.045
−0,006
(0.086)

274
0.293**
(0.117)

251
0.208***
(0.059)

405
0.160*
(0.091)

351

0.060

494

0.129
(0.087)

229

0.012

0.091
(0.089)

176
0.407***
(0.155)

167
0.296***
(0.073)

269
0.403***
(0.114)

206

0.011

494

0.065
(0.093)

229

0.005

0.068
(0.098)

176
0.435***
(0.169)

167
0.280***
(0.077)

290
−0.091
(0.147)

117

0.738
−0.140
(0.137)

0.022
(0.122)

0.059
(0.068)

269

136
0.419*** −0.251*** −0.225**
(0.098)
(0.078)
(0.123)

136

206

0.012

145

0.775

145

0.627

−0.146
(0.153)

96
−0.057*
(0.032)

367

0.544
−0.068
(0.075)

290
−0.012
(0.148)

117

0.785
−0.165
(0.165)

0.116
(0.125)

0.061
(0.090)

Notas: This table reports estimates from a regression of unit-quiz exam scores on an indicator for being assigned to a flipped classroom for
the subgroups identified in each row. All scores have been standardized to have a mean of 0 y una desviación estándar de 1 for each subject.
All estimates include the controls described in table 5. Errores estándar, clustered on classroom, se informan entre paréntesis.
***Significativo en el 1% nivel; **significativo en el 5% nivel; *significativo en el 10% nivel.

= 0.005). Próximo, we use ACT scores to proxy for students’ baseline math ability and
interés. Students who scored in the bottom quartile of the ACT (relative to their class-
compañeros) experienced no signiﬁcant eﬀects from the ﬂipped classroom, while the higher
scoring students experienced signiﬁcant positive gains.

Juntos, these subgroup eﬀects show that the ﬂipped classroom has the opposite
equity implications as proponents claim. By having a null eﬀect on the bottom of the
math ability distribution, the ﬂipped classroom exacerbated the achievement gap while
not serving women, black students, and Hispanic students.16,17

16. We also used the method suggested by Abadie, Chingos, and West (2018) to investigate the impact of the ﬂipped
classroom by how we predict students will score on the math quiz in the standard classroom. The results of
this investigation are similar to our estimated treatment eﬀects by ACT quartile: The ﬂipped classroom boosts
student performance for students whom we predict will score in the top of the distribution, but there are no
gains for those in the bottom quartile. Results are available from the authors upon request.

17. We ﬁnd no substantial diﬀerences in the student survey responses on class preparation or students’ views on

the usefulness of the videos and readings that might explain these diﬀerential eﬀects.

382

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

Notas: This figure displays the racial and baseline academic ability achievement gaps for the treatment and control groups for both
the quiz and the final exam questions specific to the experimental unit’s content. Achievement gaps are calculated by differencing
the mean standardized scores of the top and bottom ACT quartiles and the white and Black or Hispanic students.

Cifra 3. Racial and Baseline Academic Ability Achievement Gaps

Cifra 3 illustrates how the ﬂipped classroom impacted the achievement gap. El
gap in math quiz performance between white students and black or Hispanic students
es 0.263 standard deviations in the control group. The ﬂipped classroom achievement
gap is 69 percent larger, en 0.444 desviaciones estandar. Similarmente, the diﬀerence in math
quiz means between the top and bottom quartile of ACT scorers is larger in the treat-
ment group by 23 por ciento (0.838 standard deviations diﬀerence in the ﬂipped class-
room compared with 0.682 in the standard classroom). Although we ﬁnd no aggregate
eﬀects of the ﬂipped classroom on the ﬁnal exam questions related to the experimen-
tal unit, the achievement gap diﬀerences persist at similar rates. We ﬁnd a 51 por ciento
larger racial achievement gap diﬀerence and a 25 percent larger ACT achievement gap
in the ﬂipped classroom relative to the standard lecture.18

The subgroup analysis among Economics classrooms reveals few noticeable dif-
ferences, although it does appear that the ﬂipped classroom is least eﬀective for the

18.

It is worth noting that this exacerbates the achievement gap among a relatively high-performing group of stu-
dents who went through the selective West Point admissions process. In the 2017–18 admissions cycle, 25
percent of admitted applicants scored below 550 on the English section and 590 on the Math section of the
SAT while the top quartile of admitted applicants scored above 660 y 690 on English and Math, respectivamente
(West Point 2019).

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

383

Effects of Flipped Classroom Instruction

highest ACT quartile (even though there is no statistically signiﬁcant diﬀerence be-
tween the top and bottom of the ACT distribution as there is with math). It appears
that the racial achievement gap is larger in the ﬂipped classroom relative to the stan-
dard lecture for both the quiz and ﬁnal exam questions (see ﬁgure 3), although we ﬁnd
mixed results for the ACT quartiles achievement gap.

7 . CONCLUSIÓN
Several design features of the experiment mute the potential eﬀectiveness of the ﬂipped
classroom model. We conducted the experiment for three class sessions. This could
lead to an underestimation of the model’s eﬀect because it may have been disruptive
for the treatment group to switch the class format without enough time to establish
strong classroom norms. Además, we expect that instructors would improve over
time as they get used to implementing the new pedagogy. While it is a strength of our
study design that we observe instructors in both the standard lecture and the ﬂipped
aula, it means that instructors have to prepare two types of classes. This extra
preparation may lead to lower-quality classes than if they focused on one type of lecture.
Además, we did not allow faculty or students to select into the study. Those who
prefer the new pedagogy might be better instructors and students in this model, como
suggested by our survey results. Por último, additional instructor training and support could
improve the quality of implementation. Juntos, these constraints suggest that our
ﬁndings could be a lower bound for the potential impacts of the ﬂipped classroom. Nuestro
ﬁnding that the ﬂipped classroom generates substantial gains in math—but widens the
achievement gap—might also serve as a lower bound for the potential eﬀects for longer
implementations without these limitations.

Although the standardized nature of course content, teaching, and grading oﬀers
an ideal setting for a randomized control trial, West Point is a unique academic set-
ting and our results might not generalize to other postsecondary institutions. On one
mano, West Point’s selective admissions criteria and small class sizes are comparable
to selective liberal arts colleges. Por otro lado, cadets at West Point and the pre-
dominantly military faculty who teach them might diﬀer on unobservable dimensions
from students and faculty at other colleges. En particular, West Point’s emphasis on
class attendance and classroom discipline could make ﬂipped classroom instruction
at West Point more eﬀective (p.ej., through increased participation) or less eﬀective
(p.ej., if marginal cadets negatively inﬂuence their peers) than ﬂipped classroom in-
struction in other settings. Considering our results, additional research on the impact
of the ﬂipped classroom model in more traditional postsecondary settings is clearly
warranted.

We implement a clean ﬂipped classroom experiment in a setting with real stakes
and glean insights about this popular pedagogical model. The results of our experiment
show that the ﬂipped classroom can generate large learning gains in a short period of
time and that implementation quality, instructor preferences, and student engagement
likely play key roles in its eﬀectiveness. We ﬁnd substantial short-term eﬀects in math
and null eﬀects for economics. Suggestive evidence points to a few potential explana-
ciones. Primero, instructors who preferred teaching the ﬂipped classroom generated larger
efectos, suggesting that instructor interest contributes to the success of new pedagog-
ical models. Segundo, we ﬁnd higher levels of student engagement in math relative to

384

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

economics: Math instructors report higher rates of students paying attention, asking
preguntas, and working in groups and independently in class relative to the math con-
trol group and the economics treatment group. También, math students rate the video more
useful than economics students. Survey data also show that the math classrooms in-
creased student engagement and student-teacher interactions more than the economics
classrooms (perhaps those aspects are important for an eﬀective ﬂipped classroom
implementación).

Despite the short-term eﬀects in math, we ﬁnd no longer-term gains in learning and
the ﬂipped classroom exacerbates the achievement gap instead of reducing it. Short-
term gains in math are concentrated among male, white, and high-achieving students.
The ﬂipped classroom has a 69 percent larger racial achievement gap and a 23 por ciento
larger baseline, academic ability achievement gap than the standard lecture, and these
diﬀerences persist through the ﬁnal exam. Combined, these ﬁndings suggest educators
should exercise caution when implementing the ﬂipped classroom.

Even with null long-term eﬀects, educational institutions may still choose the
ﬂipped classroom model if it maintains average levels of learning, but at lower costs.
Schools can reduce costs by using the ﬂipped classroom model by hiring lower-skilled
instructors, such as teaching assistants or tutors to facilitate the ﬂipped classroom, y
paying a one-time cost to produce high-quality videos by a skilled lecturer.

EXPRESIONES DE GRATITUD
Special thanks go to the Math and Economics departments of the United States Military Academy
for participating in the study. We also thank Sandra Black, Susan Dynarski, David Figlio, Joshua
Buen hombre, Sarah Komisarow, Jonah Rockoﬀ, and seminar participants at Tufts University, el
United States Military Academy, the Northeast Economics of Education Workshop, the Asso-
ciation for Public Policy Analysis & Management, and the Western Economic Association for
helpful comments. Setren was supported by a National Science Foundation Graduate Research
Fellowship. The views expressed herein are those of the authors and do not reﬂect the posi-
tion of the United States Military Academy, the Department of the Army, or the Department of
Defensa.

REFERENCIAS
Abadie, Alberto, Matthew Chingos, and Martin West. 2018. Endogenous stratiﬁcation in random-
ized experiments. Review of Economics and Statistics 100(4): 567–580.

alperto, William T., Kenneth A. Couch, and Oskar R. Harmon. 2016. A randomized assessment
of online learning. Revisión económica estadounidense 106(5): 378–382.

Bergmann, Jonathan, and Aaron Sams. 2009. Remixing chemistry class: Two Colorado teachers
make vodcasts of their lectures to free up class time for hands-on activities. Aprendiendo & Leading
with Technology 36(4): 22–27.

Bergmann, Jonathan, and Aaron Sams. 2012. Flip your classroom: Reach every student in every class
every day. Washington, corriente continua: International Society for Technology in Education.

Berrett, Dan. 2012. How “ﬂipping” the classroom can improve the traditional lecture. Chronicle
of Higher Education, 19 Febrero.

Bettinger, Eric P., Lindsay Fox, Susana Loeb, and Eric S. taylor. 2017. Virtual classrooms: Cómo
online college courses aﬀect student success. Revisión económica estadounidense 107(9): 2855–2875.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

385

Effects of Flipped Classroom Instruction

Bowen, William G., Matthew M. Chingos, Kelly A. Lack, and Thomas I. Nygren. 2014. Interactive
learning online at public universities: Evidence from a six-campus randomized trial. Diario de
Policy Analysis and Management 33(1): 94–111.

Brame, Cynthia. 2013. Flipping the classroom. Available http://cft.vanderbilt.edu/guides-sub-
pages/ﬂipping-the-classroom. Accedido 10 Enero 2020.

Carretero, Susan Payne, Kyle Greenberg, and Michael S. Caminante. 2017. The impact of computer
usage on academic performance: Evidence from a randomized trial at the United States Military
Academia. Revisión de la economía de la educación 56:118–132.

Davies, Randall S., Douglas L. Dean, and Nick Ball. 2013. Flipping the classroom and instructional
technology integration in a college-level information systems spreadsheet course. Educativo
Technology Research and Development 61(4): 563–580.

Escueta, Maya, Vincent Quan, Andrew Joshua Nickow, and Philip Oreopoulos. 2017. Educación
tecnología: An evidence-based review. NBER Working Paper No. 23744.

Figlio, David, Mark Rush, and Lu Yin. 2013. Is it live or is it internet? Experimental estimates
of the eﬀects of online instruction on student learning. Journal of Labor Economics 31(4): 763–
784.

Findlay-Thompson, Sandi, and Peter Mombourquette. 2014. Evaluation of a ﬂipped classroom in
an undergraduate business course. Business Education & Accreditation 6(1): 63–71.

Goodwin, Bryan, and Kirsten Miller. 2013. Evidence on ﬂipped classrooms is still coming in.
Educational Leadership 70(6): 78–80.

Joyce, Ted, Sean Crockett, David A. Jaeger, Onur Altindag, and Stephen D. O’Connell. 2015. Hace
classroom time matter? Revisión de la economía de la educación 46:64–77.

Lage, Maureen J., Glenn J. Platón, and Michael Treglia. 2000. Inverting the classroom: A gateway
to creating an inclusive learning environment. Journal of Economic Education 31(1): 30–43.

Sotavento, David S. 2009. Capacitación, wages, and sample selection: Estimating sharp bounds on treat-
ment eﬀects. Review of Economic Studies 76(3): 1071–1102.

Lo, Chung Kwan, and Khe Foon Hew. 2017. A critical review of ﬂipped classroom challenges in K-
12 education: Possible solutions and recommendations for future research. Research and Practice
in Technology Enhanced Learning 12(4): 1–22.

McLaughlin, Jacqueline E., Mary T. Roth, Dylan M. Glatt, Nastaran Gharkholonarehe, Christo-
pher A. Davidson, LaToya M. Griﬃn, Denise A. Esserman, and Russell J. Mumper. 2014. El
ﬂipped classroom: A course redesign to foster learning and engagement in a health professions
escuela. Academic Medicine 89(2): 236–243.

Overmyer, Gerald Robert. 2014. The ﬂipped classroom model for college algebra: Eﬀects on stu-
dent achievement. Doctoral dissertation, Colorado State University, Fort Collins, CO.

Schaﬀhauser, Dian, and Rhea Kelly. 2016. 55 percent of faculty are ﬂipping the classroom. Leva-
pus Technology, 12 Octubre.

Schaﬀhuaser, Dian. 2016. Investigación: Video usage in ed continues ramp-up. Campus Technology,
18 Julio.

Schultz, David, Stacy Duﬃeld, Seth C. Rasmussen, and Justin Wageman. 2014. Eﬀects of the
ﬂipped classroom model on student performance for advanced placement high school chemistry
estudiantes. Journal of Chemical Education 91(9): 1334–1339.

386

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d

b
y
gramo
tu
mi
s
t

oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3

Elizabeth Setren, Kyle Greenberg, Oliver Moore, and Michael Yankovich

Supiano, Beckie. 2018. Traditional teaching may deepen inequality. Can a diﬀerent approach ﬁx
él? The Chronicle of Higher Education, 6 Puede.

Swoboda, Aarón, and Lauren Feiler. 2016. Measuring the eﬀect of blended learning: Evidencia
from a selective liberal arts college. Revisión económica estadounidense 106(5): 368–372.

West Point. 2019. Class proﬁle. https disponibles://westpoint.edu/admissions/class-proﬁle. C.A-
cessed 10 Enero 2020.

Wozny, Nathan, Cary Balser, and Drew Ives. 2018. Evaluating the ﬂipped classroom: A random-
ized controlled trial. Journal of Economic Education 49(2): 115–129.

D
oh
w
norte
oh
a
d
mi
d

F
r
oh
metro
h

t
t

pag

:
/
/

d
i
r
mi
C
t
.

metro

i
t
.

mi
d
tu
mi
d
pag
a
r
t
i
C
mi
–
pag
d

F
/

1
6
3
3
6
3
1
9
2
8
2
0
7
mi
d
pag
_
a
_
0
0
3
1
4
pag
d