Expertise in University Teaching - Ricerca sull'intelligenza artificiale specializzata al MIT

Expertise in University Teaching
& the Implications for Teaching
Effectiveness, Evaluation & Training

Carl Edwin Wieman

Universities face the challenge of how to teach students more complex think-
ing and problem-solving skills than were widely needed in the past, and how to
teach these to a much larger and more diverse student body. Research advanc-
es in learning and teaching over the past few decades provide a way to meet
these challenges. These advances have established expertise in university teach-
ing: a set of skills and knowledge that consistently achieve better learning out-
comes than the traditional and still predominant teaching methods practiced
by most faculty. Widespread recognition and adoption of these expert practices
will profoundly change the nature of university teaching and have a large ben-
eficial impact on higher education.

U niversity teaching is in the early stages of a historic transition,

changing from an individual folk art to a field with established ex-
pertise, much as medicine did 150 years ago. What is bringing about
this transition and what can we expect of it? To answer, I start with the na-
ture of expertise and how it applies to the context of academic disciplines.
In particular, I discuss how such expertise defines disciplines and how re-
search and other scholarly work plays an essential role in establishing dis-
ciplinary expertise. Then I show how recent research has established exper-
tise in university teaching: a set of instructional practices that achieve better
student outcomes than traditional teaching methods. These advances also
illustrate the essential role that disciplinary expertise has in effective uni-
versity teaching and provide perhaps the best justification for the research
university as an educational institution. Tuttavia, while disciplinary exper-
tise is a necessary part of good university teaching, it is far from sufficient:
there are many other elements of teaching expertise. I conclude by arguing
that the widespread recognition of expertise in university teaching will im-
prove both the effectiveness and efficiency of teaching by making it a more

© 2019 dall'Accademia Americana delle Arti & Scienze
Pubblicato sotto Creative Commons
Attribuzione 4.0 Internazionale (CC BY 4.0) licenza
https://doi.org/10.1162/DAED_a_01760

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

collective and coherent endeavor with better-defined standards for evalua-
tion and training.

T here is a general process by which expertise is established in any hu-

man endeavor; this applies to both academic disciplines and universi-
ty teaching. In many areas of human activity, including music, sports,
and medicine, the concept of “expertise” is well known. In these areas, there
are individuals who can consistently achieve measurably better results than
most people. Much of the research and discussion on expertise has focused
on what it is about uniquely high-performing individuals that sets them apart.
But what is the nature of expertise more generally? What are the require-
ments for associating expertise with an area of activity? And how does this
concept of expertise apply to academic disciplines and university teaching?
There is a large literature on expertise, both what it is and how it is ac-
quired. I will use the definition given by cognitive psychologist Anders Erics-
figlio, slightly paraphrased: expertise is a specific set of skills and knowledge that are
not widely shared and can be seen to consistently produce measurably better results when
applied to relevant tasks.1 Thus, for an activity to involve expertise, there must
be readily identifiable tasks, and there must be measurable outcomes. The re-
search shows that a person’s level of expertise or, equivalently, “competence
level” steadily increases with the amount of time spent in appropriate learn-
ing activities. For mature disciplines, reaching the highest levels (becom-
ing an “expert”) requires thousands of hours of practice.2 When I refer to an
“expert” here, I mean a recognized successful practitioner in the discipline;
Per esempio, the equivalent of a typical university faculty member.

From the studies of expertise across multiple fields, including my own
research looking at different academic disciplines, I argue that, in the con-
text of academic disciplines, expertise is primarily defined in terms of a set
of decisions. It is applying the skills and knowledge of the discipline to make decisions
with limited information in relevant novel contexts. The quality of those limited-
information decisions–be they which scholarly question or problem to pur-
sue, which information is relevant and which irrelevant, choosing methods of
analyses, how to structure an argument, choosing standards of evidence, or jus-
tification of conclusions–all rely on the standards of the discipline. An activity
can only exist as a recognized discipline if there are consensus standards that
are used to evaluate the quality of scholarly work (such as the quality of the de-
cisions embodied in that work) E, correspondingly, the quality of scholars in
a field (Per esempio, in academic hiring and promotion decisions). A require-
ment for the establishment of such standards is a foundation of “research”/
scholarly work that has demonstrated that, among the possible alternative

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

decisions that a person might make, there are particular choices and processes
for making such decisions that consistently achieve better results.

In some activities, particularly sports, there are clear quantitative mea-
sures of overall performance, and so the “research” proceeds rapidly, estab-
lishing which practices and training methods lead to improvements in out-
comes. In a new video game, Per esempio, the establishment of expertise in
game performance happens very rapidly. In academic disciplines, the out-
comes, and the connections between performance elements (like decisions)
and outcomes, are more complex. Then the research process proceeds more
slowly, as extensive research is needed to establish what factors do and do not
impact outcomes, and over what range of contexts and performers.

To establish levels of competence and guide improvement, it is also es-
sential to resolve expertise in a field into the set of subskills or practices re-
quired in the ultimate performance. Per esempio, rather than simply having
standards as to what constitutes well-played violin music, there are accepted
standards as to what is good fingering technique, bowing technique, and so
on that the “research” by music teachers has shown are important for achiev-
ing the ultimate goal of good music. Così, there are standards that guide the
learner in practicing and mastering that subskill, even while they are doing
other things wrong and good music is not being produced. In academics, come
standards for subskills would apply to the outcome of the decisions listed
above, such as choice of question or sources of evidence. Making such deci-
sions in an expert way involves both having the relevant knowledge and hav-
ing the reasoning skills to guide when and how that knowledge is used. In to-
tal, these standards for subskills, encompassing appropriate knowledge and
its use to make decisions, largely define expertise in a discipline. With suffi-
cient practice, some of these decisions become automatic, carried out with lit-
tle conscious thought, thereby increasing the speed of the process.

The role of research in establishing expertise is illustrated by the field of
medicine. In the 1400s, the definition of what it meant to be a good doctor
was quite arbitrary and varied according to individual idiosyncrasies. Anyone
and everyone could believe, and announce to the world, that they were a good
doctor, even though different doctors employed a wide variety of practices. UN
similar situation exists today with regard to education; almost everyone who
has been to school, let alone taught a class, believes that they are an expert, In
that their opinion has equal or greater weight as that of anyone else.

Over the subsequent centuries, medical research led to the establish-
ment of knowledge, principles, and methods that produced consistently bet-
ter results. A practitioner who knew and applied these produced better out-
comes (healthier, more long-lived patients) than those who did not, making

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

it possible to set objective standards for who was a competent doctor. This in-
cluded standards about the components of expert practice such as washing
hands between patients, knowing which diagnostic tests to use, and prescrib-
ing the most effective treatments. The transformation of medicine illustrates
how fields change as a research base is established, leading to the recognition
of expertise in the field. This establishment of research-based medical exper-
tise led to changes in the training and conduct of medicine, with resulting im-
provements in both outcomes and the rate of further progress. The transition
of alchemy into the modern discipline of chemistry is another example illus-
trating how an academic discipline with expertise develops following the cre-
ation of an adequate research base.

Teaching has traditionally not been an area for which well-defined exper-
tise exists; it is more often characterized as an “art” wherein each individual
is encouraged to choose their preferred style. While there has been a general-
ly accepted goal–learning–what that means and how it can be measured has
been ill-defined and variable. It is striking to read the many recent OECD (Or-
ganisation for Economic Co-operation and Development) reports on improv-
ing the quality of university teaching and see that none of them actually de-
fine teaching quality or how it could be measured. “Good” teachers are often
described in terms of personal characteristics like “enthusiasm,” “concern
with students,” and “interest in their subject.” Judgments of teaching qual-
ity have traditionally depended largely on individual preferences, much like
the judgment as to whether a painting is attractive or not, or whether a person
is likeable. At the level of the institution or academic department, efforts to
“improve teaching” often focus on the curriculum: what topics are covered in
what order. Research on learning, Tuttavia, implies that such curricular choic-
es play at best a secondary role in determining meaningful student learning
outcomes, particularly learning to think more like an expert in the discipline.
The lack of agreed-upon standards for teaching quality allows everyone to
consider themselves to be a good teacher by some standard, and most do.

Research during the past few decades has changed this situation for uni-
versity teaching, although this change has yet to be widely recognized. These
advances in research now make it possible to define expertise in university-
level teaching and, correspondingly, define teaching quality in an objective
expertise-based manner. The research comes from a combination of studies
in cognitive psychology and the science of learning, studies in university sci-
ence and engineering courses, E, most recently, from brain research. Questo
includes hundreds of laboratory and classroom studies involving controlled
comparisons of different teaching methods, primarily, but not exclusively,
measuring student learning.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

Much of the classroom research is the result of the relatively new field of
“discipline-based education research” (DBER), which has developed over the
past few decades.3 This research focuses primarily on undergraduate learn-
ing of the science, technology, engineering, and mathematics (STEM) disci-
plines at research universities, and is carried out by faculty in the respective
disciplines (physics, biologia, computer science, so on).4 This is distinct from
the educational research that is carried out in schools of education, che è
largely confined to the K–12 level.

The standards of DBER have rapidly evolved, and different disciplines are
still at different stages of progress in this evolution. Not long ago, such univer-
sity education “research” consisted of instructors trying some change in their
teaching of a course and measuring the impact in some idiosyncratic way, pri-
marily how much the students liked it. Now, quality DBER, which is what I
am discussing here, is similar to medical research. It requires controlled com-
parisons of different ways to teach particular material, and the impacts are
measured using validated, often published, and widely used tests that probe
apprendimento. Research protocols are similar to those for other human-subjects re-
search and have the same institutional review.

DBER has led to new types of assessments of learning, new teaching meth-
ods, and comparisons of learning achieved with different methods of instruc-
zione. The research has explored the importance of many different factors for
student learning, course completion, E, occasionally, student retention in
a major. The teaching methods that have been found to be the most effective
are well aligned with cognitive psychology research on learning, sometimes
by intention and other times not.5 This alignment is particularly evident in
the research on teaching expert thinking, which has illustrated the need for
explicit practice of the mode of thinking to be learned along with guiding
feedback.

The assessments of learning in DBER that have been the most sensitive and
impactful are “concept inventories.” Such inventories are carefully developed
to probe the extent to which students can apply relevant disciplinary concepts
like an expert in the field to novel situations appropriate to the course content.
Their primary use is to measure the effectiveness of the teaching in the class
as a whole, rather than the learning of the individual students per se. Such in-
ventories now exist for material covered in a number of standard introducto-
ry science and math courses and a few upper-level science courses. These pro-
vide researchers with good instructor-independent measures of learning that
can be widely used, and hence allow widespread, carefully controlled com-
parisons of different teaching methods. These assessments are based on the
unique disciplinary frameworks for making decisions that experts use, Piuttosto

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

than based on remembering pieces of knowledge or a memorized procedure.
As such, learning to do well on these assessments of “expert thinking” is more
sensitive to instructional practices than typical exam questions and less sen-
sitive to “teaching to the test.” These kinds of assessments have become a
uniquely valuable tool for research on the relative effectiveness of different
types of university teaching, but for practical reasons, they only measure a
subset of the relevant expert thinking. There are other aspects that must be
measured in different ways, including things like deciding on choices of pos-
sible solutions or designs, recognizing the range of real-world situations in
which the discipline can be useful to understand and predict important phe-
nomena, and the learner deciding they can master and enjoy working in the
discipline.

Researchers also look at more conventional outcomes, such as failure rates
and course and exam grades, but those are more sensitive to the character-
istics of the incoming students and the idiosyncrasies of individual instruc-
tori, and thus are less reliable measures. Nevertheless, they still have reason-
able validity if there are consistent standards and the instructor is careful in
the exam construction, because of the degree of standardization of the under-
graduate STEM curriculum, textbooks, and instructional goals across univer-
sities. Unfortunately, this is not true for many STEM exams that, often unin-
tentionally, primarily test the student’s memory of basic terminology, facts,
and procedures.

DBER in university STEM courses is a relatively young field and is not wide-
ly known. It has primarily been carried out in the United States and funded
by the National Science Foundation. It tends to be published in specialized
journals (Physical Review Physics Education Research, CBE–Life Sciences Education,
Chemistry Education, and Journal of Engineering Education, among others), con
an occasional article published in Science or Proceedings of the National Academy
of Sciences. There is limited awareness of DBER within the broader university
faculty and administration, with the level of knowledge varying significant-
ly by discipline. With a few exceptions, DBER is also little-known outside of
North America. Some recent reports and reviews have attempted to synthe-
size and disseminate the findings of DBER and its implications for improving
university teaching.6

DBER has established that there are particular principles and practices that
consistently achieve better student outcomes than the traditional didactic
lecture and high-stakes exam. This has typically been shown through exper-
iments involving controlled comparisons. These effects are sufficiently large
Quello, when one takes incoming student preparation into account by measur-
ing learning gains rather than just outputs, the choice of teaching practices

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

results in larger differences than any other identified variables associated
with the teacher (for instance, rated quality as a lecturer) or the students. IL
results have been replicated within and across instructors, istituzioni, cours-
es, and disciplines.7

Such results have been shown in all the disciplines in which extensive
classroom studies have been carried out, including all science and engineering
disciplines at the university level and, to a lesser extent, mathematics. There
have been some studies in other types of higher education institutions and a
few recent, small studies in the social sciences.

It would be worthwhile to carry out similar controlled comparisons of
learning in a broader range of disciplines such as history and classics. There
are theoretical reasons to think that the same teaching methods would likely
also work well in such fields, if properly adapted. The methods that have been
consistently effective reflect fundamental mechanisms for learning from cog-
nitive psychology (Guarda la figura 1), particularly for learning to think like an ex-
pert in the discipline, as mapped onto the particular course and student popu-
lation.8 The DBER that has produced the biggest gains in learning has involved
looking at the decisions that students make in solving problems after receiv-
ing traditional instruction and how they differ from those of scientists, E
then designing educational activities that involve the students explicitly prac-
ticing making such decisions with feedback. Sam Wineburg has identified
some key elements of historian expertise, including how historians determine
the credibility of historical artifacts and what conclusions they decide they
can draw from them, and how their thinking in this regard differs from col-
lege students who have taken a history course. It seems like these aspects of
historian thinking could be directly incorporated into the corresponding
research-based methods developed in STEM, likely with corresponding im-
provements in learning.

In this discussion, I have been careful to distinguish university teaching
from teaching at the K–12 level. In The Cambridge Handbook of Expertise and
Expert Performance, psychologist James Stigler and education scholar Kevin
Miller present an excellent discussion of the challenges faced in establishing
and defining K–12 teaching expertise in the United States.9 As they have dis-
cussed, there are a number of confounding variables outside the control of the
K–12 teacher, most notably the local context, that make K–12 teaching hard-
er to characterize and harder to study. It is useful to contrast the K–12 con-
text they describe with teaching in research universities where most DBER
has been carried out. Variables such as classroom behavior, the subject mat-
ter mastery of the teacher, the scheduling of teaching and assessment activ-
ities, and the extent of variability in the student backgrounds are all major

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

Figura 1
Principles and Practices of Effective Teaching

Student variation

Disciplinary
expertise

Prior knowledge
& experience

Motivation

Brain
constraints

Apprendimento
through practice with
feedback

Tasks/questions
with deliverables

Social learning

Implementation

Note: This figure represents the full span of research on the principles and practices in-
volved in learning to make good decisions in a specific disciplinary context. At the center
are the essential components of learning. This represents the intense practice of the spe-
cific elements of thinking to be learned, ideally the decision-making skills that experts in
the subject use in relevant situations, combined with feedback that guides improvement
in that thinking. The top row of boxes represents factors that enable and facilitate this
learning process. Much of the apparent variation across the student population comes in
through the motivation and prior knowledge boxes, both of which depend heavily on the
learners’ prior experiences. The two boxes in the bottom row represent consistent ele-
ments in the implementation of highly effective teaching practices.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

issues in K–12, but these are much smaller factors at the university level (even
though nearly all university teachers complain about the level and uniformi-
ty of the preparation of their students). The U.S. K–12 context is also highly
variable across schools, districts, and states, and these differences play a large
role in the educational practices and assessment. In contrasto, the context of
university teaching is far less variable: relative to K–12, there is a high degree
of standardization of the curriculum, the textbooks, the student populations
and behavior, the instructional settings, the subject mastery of the instruc-
tori, and the desired learning outcomes. This makes the classroom research
at the university level far simpler and cleaner, and it provides more definitive
results than research in K–12 teaching. In the future, greater K–12 standard-
ization through vehicles such as the Common Core State Standards Initiative
and Advanced Placement courses might provide more K–12 uniformity. Sti-
gler and Miller do propose three “teaching opportunities” that they believe
would be the characteristics of expert teachers, if sufficiently clean research
results could be obtained; these overlap with what I present below.

W hen expertise is first being established in a field, the distinctions

as to different levels of competence are relatively crude. One can
become an “expert,” a top performer, merely by recognizing ba-
sic decisions that need to be made and, in those decisions, accounting for the
basic factors that have been shown to be most relevant. As university teach-
ing is a new area of expertise, one can achieve relatively high levels of mas-
tery merely by using the basic principles and practices that have demonstrat-
ed improved learning. The description of expertise here is limited to this rel-
atively coarse level. As any discipline matures, more complexity and nuance
are seen to result in higher quality decisions, and thus more subtle factors
become recognized as elements of expertise. This will eventually happen in
teaching.

Before I can talk about what constitutes expert teaching, I need to define
the intended learning goals that such expert teaching will reliably achieve.
Often, the stated goals (or “objectives”) of courses are expressed in terms of
“understanding” or “appreciating” various topics. From extensive discus-
sions with faculty members as to what they mean by such vague statements,
I claim that the goals of the great majority of university STEM courses can be
summarized as: teaching students to think about and use the subject like a
practitioner in the discipline, consistent with the student’s background and
level. In practice, this means making relevant decisions and interpretations
using the reasoning and knowledge that define expertise in the discipline. Of
course, the level of sophistication with which the students might learn to do

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

that and the complexity and range of the contexts in which they are capable of
making such decisions will vary widely according to the course. For the dedi-
cated fourth-year chemistry major, that decision might be how best to synthe-
size a molecule in an industrial setting, while for a major from another disci-
pline taking their one required chemistry course, it might be deciding not to
pour hydrochloric acid down the drain or deciding not to invest in a company
that claims it has a process for turning seawater into gold. But “thinking like a
chemist” is needed for all these decisions. Così, I am taking the basic goal of
most university courses as having students learn to think more like an expert
in their respective discipline.10

The most basic principle that every teacher should know about teaching
this sort of thinking is that the brain learns the thinking it practices, but lit-
tle else. To have students learn to recognize relevant features and make rel-
evant decisions more like an expert in the field, they must practice doing ex-
actly this. The longer and more intense the practice, the greater the learn-
ing. There is a biological origin to this requirement, as such intense mental
practice modifies and strengthens particular neuron connections, and the
new thinking capabilities of the learner reside in this “rewired” set of neu-
rons. There is much research on how the brain changes the way it organizes
and accesses relevant information as it learns, and on the connection between
the functional and structural changes that occur in the brain during extended
learning of expertise.11

The basic principle that people learn from practice with appropriate feed-
back is placed at the center of Figure 1. To my knowledge, practice and feed-
back are part of all research-based instruction that shows significantly better
learning outcomes than the traditional lecture. These are also the two most
basic elements of “deliberate practice,” which has independently been found
to be essential for the acquisition of expertise.12 The first element in this con-
text means having the learners actively and intently practicing the thinking to
be learned. One particularly important and often overlooked feature in teach-
ing is that thinking like an expert is primarily about making particular deci-
sions. So, the learning task must involve the learners actually making relevant
decisions. Too often, instruction only involves the teacher modeling a solution
process by telling students the decisions that the expert has made. The dif-
ferences in learning between a student being told the desired outcome of a
decision versus having the student make the decision, even if incorrect, E
then reflect upon the outcome of their decisions while supported by instruc-
tor guidance are profound.13 These differences are easy to appreciate if you
think about learning to find your way through a strange city. If you go between
two locations by simply following the directions for each turn provided by a

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

person or mapping program, you will be incapable of telling another person
how to do it or reproduce it on your own. If you had to form a mental map of
the city and explicitly decide on the turns, you will have learned far more. In
this case, you practiced making decisions and strengthened neuron connec-
tions in the necessary way to learn, and it does not matter if some of those
turns were wrong, and you had to revise your route: you still learned better the
correct decisions. This also carries over to your learning better how to trans-
fer your knowledge to a new context, such as going between new locations
or dealing with road closures. The same principles apply to learning problem-
solving decisions in a discipline.

Effective teaching is about first designing learning activities that have the
student carrying out tasks that require them to make decisions using the spe-
cific reasoning processes, including the associated requisite knowledge, to be
learned. The second element is good feedback, which means feedback that is
timely, specific, nonthreatening, and actionable.14 To be able to provide such
feedback requires that the instructor monitor the learner’s thinking in some
modo, and then use that information to provide feedback to guide the improve-
ment in that learner’s thinking (often labeled as “formative assessment”).
Under this broad general principle of practice with feedback, there is a de-
tailed set of factors that have been shown to play an important role in sup-
porting this learning process.15 These are illustrated in Figure 1. Each of the
boxes in the upper row represents a well-studied principle involving estab-
lished mechanisms of learning. Good instructional design incorporates these
principles into the design of the practice tasks and the types of feedback pro-
vided. The two boxes in the bottom row represent research on how best to im-
plement these in instructional settings. If and how the instruction incorpo-
rates the best practices represented in all of these boxes is a measure of teach-
ing expertise.

Disciplinary expertise. Embedding expertise in the subject into the instruc-
tional activities is a fundamental requirement. This expertise includes recog-
nizing what decisions need to be made in relevant contexts, along with the
tools, reasoning, and knowledge of the discipline to make good decisions.16 In
this regard, good instructional tasks should directly reflect the standards that
define expertise in the discipline discussed above, as mapped onto the context
of the specific course being taught. This involves many different decisions,
but an example of the most general and basic is, when confronted with an au-
thentic problem/question and context, deciding what the key features and in-
formation are, and what information is irrelevant to solving the problem. Ar-
tificially constrained “textbook type” problems remove practice in this criti-
cal decision skill.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

Motivation. Serious learning is inherently hard work that involves pro-
longed strenuous mental effort. The motivation to engage in that effort plays
a large part in the learning outcomes. Motivation is obviously enhanced by
making a subject interesting and relevant to the learner, which often means
framing the material in terms of a meaningful (to the learner!) context and
problem that can be solved.

A less obvious element in motivation is having a “growth mindset,” the
learners’ belief that they can master the subject and a sense of how to attain
that mastery, a belief that can be powerfully affected by both prior experienc-
es and teacher behaviors.17 Too often teachers fail to recognize the impact of
the various messages they convey through what they say or how they grade.
Per esempio, an exam that measures all of what students should have learned
and only that, compared with the more typical exam that focuses on the most
challenging material that will provide the best differentiation between stu-
dents, send very different signals to students. The first shows them all of what
they are learning and is motivating, while the second leaves many students,
for example those who only get a 50 percent score after intensive study, con un
demotivating sense of failure and frustration, even if that is the class average.
Prior knowledge and experience. To be effective, instructional activities must
match with and build upon what the student already knows and believes
about the subject and how to learn it. Research has shown that it is important
for effective instruction to recognize and address even very specific aspects of
the learners’ thinking about particular topics, such as whether a student be-
lieves that heavier objects fall more rapidly than lighter objects when teach-
ing introductory physics.

Both prior knowledge and what does and does not motivate students are
highly dependent on their prior experiences. Hence, these are the areas where
most of the observed variations in the student populations are apparent. IL
expert teacher will recognize it is inadequate to ask students what they know
or come to conclusions based on the syllabi of prior courses the students have
taken. Instead they will measure what the students know and can do, initial-
ly and ongoing through the course. They will then optimize learning by ad-
justing their instruction to match best the characteristics of their student
population.

Brain constraints. The next box, constraints of the brain, refers to 1) the lim-
ited capacity of the short-term working memory of the brain (five to seven
new items, far less than introduced in a typical class session) and its well-stud-
ied impacts on learning; E 2) the processes that hinder and help long-term
retention of information. The limited capacity of working memory means
that anything peripheral to the desired learning that attracts the learner’s

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

attention will reduce the desired learning. This includes new jargon, attractive
images, or even amusing stories or jokes. The biggest problem with long-term
retention is not in remembering material in the first place; Piuttosto, it is correct-
ly retrieving it later after additional material has been learned. That new ma-
terial interferes with the retrieval process. To avoid this interference, as new
material is learned, it needs to be intermingled with the recall and application
of old material. This is not the usual practice in STEM courses wherein novice
teachers cover the topics in a strict chronological order.

The two boxes at the bottom of Figure 1 represent key elements for the im-

plementation of research-based teaching:

Tasks/questions with deliverables. To ensure that students are practicing the
desired thinking, they need to be given tasks or questions that explicitly re-
quire that thinking. Explicit deliverables achieve engagement in the task and
provide essential information to the teacher for giving effective feedback. For
esempio, in a genetics class, students would consider the blind fish in Mexi-
can caves. They would be asked to consider what they could decide about the
number of genes containing the blindness mutation from the distribution of
blindness in the offspring of true-breeding lines of fish bred from lines in two
different caves. In a large class (two hundred to three hundred students), IL
instructor would have the students answer using a personal response system
(PRS), followed by small-group discussion (that the instructor and TAs moni-
tor) and a second vote. In a smaller class, students would have to write out their
prediction with the reasoning, to be turned in for participation credit, possibly
in addition to the PRS questions. In a physics class, they would be given a prob-
lem to solve for a particular physical situation, such as predicting how much
electricity could be produced from a hydroelectric plant: the first step would
be to write out which physics concepts are most relevant to solving the prob-
lem and why, to be turned in later and minimally graded; the instructor and
TAs would circulate and read students’ responses during class. In a large class,
this could be followed with a PRS question testing them on their choices. In all
of these cases, there should be follow-up homework questions, and it should
be explicit that there will be quite similar questions on future exams.

Social learning. Interacting with peers during the learning process is a valu-
able and commonly used facilitator of learning.18 It supports learning in mul-
tiple ways. Students get timely knowledge and feedback from their peers, Essi
learn the standards of discourse and argument of the discipline, and they de-
velop metacognitive skills through their critique of others’ reasoning and
hearing others question their own. Finalmente, there are unique cognitive process-
es that are triggered by social interactions that produce learning. Even antici-
pating that one will teach a peer about a topic has shown to improve learning

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

over just studying the topic. E, Ovviamente, such group activities provide op-
portunities for the students to learn collaborative skills. Important elements
of teaching expertise are to know how to avoid the potential pitfalls of group
lavoro, how to set and monitor norms of behavior, and how to structure the
group activities to achieve all of the potential benefits.

The set of factors and practices represented in Figure 1 largely determine
learning outcomes at the university level for the disciplines and institution
types in which they have been tested. There are many examples where very
experienced faculty have changed their teaching practices to incorporate
these principles and practices, usually moving from lecture to research-based
instruction, and achieved substantial improvements in student learning out-
comes. Research is ongoing as to how best to take these factors into account in
the design and implementation of the learning process across the full range of
disciplines, topics, and students. Tuttavia, the relevance and benefits can be
understood in terms of established general mechanisms of learning, and thus
it is likely that they will apply across nearly all higher education settings and
academic disciplines.19

If a teacher is applying these practices in a discipline in which they have
not been studied, the respective disciplinary standards of expertise and asso-
ciated decisions must provide the foundation of the educational practice tasks
that learners carry out, as well as the feedback they receive. This emphasizes
the need for every good university teacher to have a high level of disciplinary
expertise.

In summary, the experimental study of how learning takes place and how
best to facilitate it in university teaching has provided a rich body of evidence
establishing the basis of expertise in teaching. Research consistently shows
better student outcomes compared with lectures when students are fully en-
gaged in challenging tasks that embody expert thinking and they receive guid-
ing feedback: the principles represented in Figure 1. This success is the basis
for my claim that expertise in university teaching exists. An expert teacher
will be aware of these principles and use suitable research-tested practices to
incorporate all of them into their instruction.

In one respect, it is somewhat surprising that the research results are so
consistent.20 As in every discipline, there are countless ways for a novice to
do such complex tasks poorly, even if trying to follow best practices. These
research-based teaching practices are regularly being adopted by faculty with
little teaching expertise, usually, though certainly not always, to good effect.
I believe that a likely reason for this consistency is that research-based teach-
ing is, to a substantial extent, self-correcting. In nearly all forms, it provides
opportunities for the instructor to know what the students are thinking and

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

struggling with–far better opportunities than instructors get when lecturing.
When instructors are first adopting these methods in even modestly informed
ways, they almost always comment on how much better they now understand
student thinking and difficulties compared with when they were teaching by
lecturing, and how this new understanding of student thinking is changing
their teaching. These new insights allow them to recognize and correct weak-
nesses in their instruction, thereby improving learning.

Although university teaching expertise can now be defined, it is not wide-
ly known and practiced. Again, the situation with university teaching is like
medicine in the mid-1800s. Although research had established a basis for sci-
ence-based medical practice, many “doctors” were unaware of that science.
Their practice was based primarily on tradition and individual superstitions
with no accepted standards. That changed during the late 1800s and early
1900S. There is reason to hope for a similar transition in university teaching.

T he establishment of expertise in teaching has implications for the

training, evaluation, and cultural norms for how teaching is carried
fuori. In every discipline, the relevant standards of expertise play a large
part in the practice and training in the discipline. Once there are well-defined
and generally accepted standards of expertise, these provide standards on
which to base both evaluation and training. This includes standards for being
certified as competent, either formally as in medical or legal licensure, or in-
formally as in the process of review of scholarly work for publication or judg-
ing the qualifications of faculty job applicants. In the case of university teach-
ing, a teacher now can, and should be, evaluated on their level of teaching ex-
pertise: how familiar they are with the principles and practices represented in
Figura 1 and to what extent they use these in teaching. Training needs to pro-
vide them with this expertise.

Evaluation of teaching quality at the university level has long been problem-
atic. Currently, the dominant method is student course/instructor evaluation
surveys. There are obvious problems with such evaluations, as well as some par-
ticularly compelling recent studies showing substantial gender bias.21 As I have
written elsewhere, the basic requirements for any good evaluation system are:

• Validity. Results correlate with the achievement of the desired student
outcomes and allow meaningful comparisons of quality across differ-
ent instructors and departments.

• Fairness. Only depends on factors under the instructor’s control.
• Guides Improvement. Provides clear guidance as to what should be done

to improve.22

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

Student course evaluations fail badly at meeting any of these criteria. Most
important for this discussion, they have been clearly shown to fail at both re-
flecting the extent of expert teaching practices being used and reflecting im-
provements in learning.

Tuttavia, it is now possible to evaluate teaching based on standards of ex-
pertise. One example of this is the Teaching Practices Inventory (TPI) devel-
oped by Sarah Gilbert and me (see Appendix I).23 It is a survey that can be
completed quickly (about ten minutes per course) and reflects nearly all the
decisions that an instructor makes in designing and teaching a course. It pro-
vides a detailed objective characterization of most of the instructional prac-
tices used in a course and, correspondingly, the extent of use of research-based
effective practices. It is not perfect; it does not show the effectiveness with
which these practices are being used. It is analogous to measuring if doctors
are washing their hands between patients, but not how well they are washing.
We and others have seen that this level of measurement is sufficient to easi-
ly distinguish between the different levels of teaching expertise present in a
typical sample of university science faculty. The TPI shows a high degree of
discrimination across a typical sample of university faculty, with the highest
scoring faculty also having very high measures of student learning outcomes.
TPI results allow meaningful comparisons to be made across faculty, depart-
menti, and institutions.

The use of such expertise-based evaluation of teaching would make it
more like the evaluation of research, allowing institutions to include teach-
ing both in their evaluation and incentive systems in a far more meaningful
and intentional way than is currently possible. It would also make it straight-
forward to set clear criteria for the level of teaching competence expected for
new faculty hires and for promotion and tenure decisions.

E ffective training of teachers, similar to good training in any area of ex-

pertise, involves practicing the relevant thinking and actions in au-
thentic contexts, along with feedback to guide improvement. As in ac-
ademic disciplines, the most important part of training in teaching is to prac-
tice the relevant decision processes, recognizing what information is most
important to guide those decisions and using it accordingly. This will require
training that is both more extensive and more targeted than most existing
university teacher training programs.

The list of elements that needs to be covered in training university teach-
ers reflects all aspects of teaching a course and all the principles represented
in Figure 1. This may seem overwhelming compared with what is now typ-
ical, but it is small compared with the training faculty received to become

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

experts in their disciplines. I have seen that faculty can reach a respectable
level of teaching expertise in something in the range of fifty hours of training;
less time than is required to complete most university courses.24 That is suffi-
cient to allow faculty members to switch from teaching by traditional lecture
and exams to research-based methods and achieve good results. Ovviamente,
this small amount of time (fifty hours) required to be reasonably competent
in teaching, compared with the thousands of hours required for high com-
petence in a mature discipline, is a reflection of the immaturity of the field
and the current level of expertise. As the level of teaching expertise increases,
the standards of competence and corresponding expectations of training and
quality will likely also increase.

I should emphasize that it does not require any additional time to teach us-
ing these new research-based methods instead of traditional teaching; it only
requires time to learn how. But in my experience, nearly all faculty that suc-
cessfully adopt these methods find that it makes teaching a far more enjoy-
able and rewarding activity. Consequently, many of them voluntarily choose
to spend more time on teaching than they had previously.25

The typical university teacher training program is too unfocused, as it is
usually designed to serve faculty from all disciplines at the same time. As with
the specificity needed for training of any type of expertise, effective develop-
ment of teaching expertise will require training programs that focus on the
teaching of the particular discipline and student population that the facul-
ty member will encounter. While the principles are general, it is a very large
step from them to knowing how to apply them to teaching a specific disci-
pline and level.

One training option is to have an individual “coach,” an approach suc-
cessfully used in many areas of expertise. Such a coach for university teach-
ing would have expertise both in the relevant discipline and in teaching in that
discipline, and would be well informed about the student population and the
other important contextual constraints. The coach would individually review
the trainee’s instructional activity designs, observe their implementation in
class, and provide feedback to guide improvement. A vital skill is also know-
ing the way things can fail, and help the trainee anticipate and avoid such fail-
ures. The use of such disciplinary teaching coaches has been shown to be an
effective model in the Science Education Initiative (SEI; see Appendix II). IL
SEI provided funding to departments to hire disciplinary experts, typically
new Ph.D.s, with a strong interest in teaching, who were then trained in the
research on teaching and learning and implementation methods, and on how
to work with faculty to support and coach them in transforming their teach-
ing. “Master-apprentice” training involving a novice teacher team-teaching a

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

course with an experienced expert teacher faculty member captures most of
the same elements and has also been shown to be effective.26

T here is a fundamental change in the social culture of a discipline when

it develops widespread recognition of expertise, a change that we can
expect in university teaching in the coming years. The establishment
of recognized expertise in a discipline enables increased collaboration/collec-
tive work and building upon prior work. When a field is recognized as an area
of expertise, like physics, chimica, or history, that means there is a com-
monly accepted set of standards and principles, along with accompanying
common language, for discussion. This commonality makes it both possible
and desirable to share ideas and methods and pursue collaborative projects,
as well as have disciplinary conferences and journals. In contrasto, teaching at
the university level is now widely seen as an isolated activity, with faculty in a
department almost never coming to view each other’s classes and seldom dis-
cussing or collaborating on teaching activities or methods. This contrast in
culture is directly related to differences in the level of recognized expertise.

It can be understood by considering the hypothetical situation of a phys-
icist whose office is in a building otherwise occupied exclusively by ancient
poetry scholars. There would be little value in the physicist going and talking
with those faculty to discuss ideas about physics, or to find new ideas for ex-
perimental designs (and vice versa, if it were a poetry scholar exiled to the
physics building). Assuming no Internet, the physicist would sit at a desk try-
ing to invent everything in isolation. But that same physicist, if located in a
building full of physicists, would be engaged in peer discussions about scien-
tific ideas and methods, gaining new information and insights and making far
more progress as a result. These physicists would be pursuing their own spe-
cific goals, but within a commonly accepted framework of principles, knowl-
edge, and standards: the core of physics expertise that facilitates discussion
and sharing for mutual benefit. This framework supports interaction and
sharing of ideas while still allowing room for identifiable individual contribu-
zione, essential components of every academic discipline.

Teaching is currently seen as a matter of individual taste and style. Each time
faculty members teach a new course, they usually design it largely from scratch,
at best taking small elements from previous offerings of the course at their insti-
tution and nothing from other institutions. This perception of teaching as a sol-
itary activity is encouraged by the institutional policies for how teaching is allo-
cated and evaluated. Each individual course is typically assigned to an individu-
al faculty member who then has full responsibility for all aspects of that course,
with very little oversight or expectations as to what will be taught and how.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

The recognition of expertise in university teaching will go hand-in-hand
with it becoming a more collective enterprise within departments and institu-
zioni, much as is the case for scholarly work in the disciplines. I observed this
in the UBC Science Education Initiative.27 There were far more frequent and
substantial discussions about teaching among the faculty in a department af-
ter a number of the faculty became moderately expert. This socialization of
teaching will in turn make teaching more efficient and effective. In scholar-
ly research, by building on past work, an individual can accomplish far more
than if they had to invent everything on their own. As practices established
through DBER have spread, there have been early examples of this happening
for teaching in some disciplines. While many elements of expert teaching are
the same across disciplines, it is likely that socialization of teaching will still
be largely confined to the existing disciplinary boundaries. That is because of
the large role that the disciplinary expertise plays, including student knowl-
edge and beliefs about the discipline, in the design and implementation of ed-
ucational activities.

T he lecture method that dominates university teaching has remained

much the same for hundreds of years. The concept of education
through an expert relaying information to a room full of novices pre-
dated the printing press, but to a large extent remains the norm today. IL
treatment of teaching as an individual art form has shaped its practice and
evaluation. This is in striking contrast to the nature of the academic disci-
plines, which have changed and advanced enormously. These medieval meth-
ods of teaching are now confronting the challenges posed by the increased
complexity of thinking that it is desirable for students to learn, and the great-
ly increased numbers and diversity of students that need a good university ed-
ucation. The acquisition of basic information is now of limited value, while
complex reasoning and decision-making skills that can be broadly applied
have high value in many aspects of modern society.

The establishment and recognition of teaching expertise has far-reaching
implications. Much as happened in medicine as it moved from its medieval
roots to modern, research-based methods, the expertise established by these
research advances in teaching provide a standard for the quality of practice,
hiring, evaluation, and training. The adoption of such standards will result in
immediate and ongoing improvements in educational effectiveness. The es-
tablishment of such consistent standards also enables the conduct of teaching
in a more collective way, using and building on previous work. This promis-
es to improve both the effectiveness and efficiency of instruction. While high-
er education is facing many challenges, the rise of teaching expertise offers a

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

path to a dramatic improvement in how it pursues its educational mission.
This would be a historic change, and while such changes never come easily,
it would provide broad societal benefits. As well as enhancing the education-
al value provided by universities, it would more clearly demonstrate their
unique educational contribution.

Many examples of teaching activities that incorporate these principles in various dis-
ciplines are given in Appendix III, accessible at http://www.amacad.org/daedalus/
teachingexpertise.

author’s note

I am pleased to acknowledge support for this work from the National Science
Foundation and the Howard Hughes Medical Institute and many valuable dis-
cussions with Dan Schwartz and the members of the Wieman research group.

about the author

Carl Edwin Wieman, a Fellow of the American Academy since 1998, is Pro-
fessor of Physics and Professor of Education at Stanford University. He is the
author of Improving How Universities Teach Science: Lessons from the Science Education
Initiative (2017) and has recently published in such journals as Journal of Educa-
tional Psychology, Physical Review Physics Education Research, and Biochemistry and Mo-
lecular Biology Education.

endnotes

1 Anders Ericsson and Robert Poole, Peak: Secrets from the New Science of Expertise (Nuovo
York: Eamon Dolan/Houghton Mifflin Harcourt, 2016); and K. Anders Erics-
figlio, Ralf Th. Krampe, and Clemens Tesch-Römer, “The Role of Deliberate Prac-
tice in the Acquisition of Expert Performance,"Revisione psicologica 100 (3) (1993):
363–406.

2 Ibid.

3 Susan R. Singer, Natalie R. Nielsen, and Heidi A. Schweingruber, eds., Discipline-Based
Education Research: Understanding and Improving Learning in Undergraduate Science and En-
gineering (Washington, D.C.: National Academies Press, 2012).

4 In what follows, I use the label “university” to refer to research universities: quelli
large institutions with substantial numbers of undergraduate and graduate de-
grees, conventional academic departments, substantial programs of scholarly
lavoro, and so on.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

5 Daniel L. Schwartz, Jessica M. Tsang, and Kristen P. Blair, The ABCs of How We Learn

(New York: W. W. Norton & Company, 2016).

6 Singer et al., Discipline-Based Education Research; President’s Council of Advisors on
Science and Technology, Engage to Excel: Producing One Million Additional College Grad-
uates With Degrees in Science, Tecnologia, Engineering, and Mathematics (Washington,
D.C.: Executive Office of the President, 2012); and Scott Freeman, Sarah L. Eddy,
Miles McDonough, et al., “Active Learning Increases Student Performance in Sci-
ence, Engineering, and Mathematics,” Proceedings of the National Academy of Sciences
111 (23) (2014): 8410–8415.

7 Freeman et al., “Active Learning Increases Student Performance in Science, Engi-

neering, and Mathematics.”

8 See also Schwartz et al., The ABCs of How We Learn.
9 K. Anders Ericsson, Neil Charness, Robert R. Hoffman, and Paul J. Feltovich, eds.,
The Cambridge Handbook of Expertise and Expert Performance, 2nd ed. (Cambridge:
Cambridge University Press, 2018).

10 A notable exception is the typical service course for nonmath majors taught by

mathematics faculty.

11 Ericsson et al., The Cambridge Handbook of Expertise and Expert Performance.
12 Ericsson and Poole, Peak; and Ericsson et al., “The Role of Deliberate Practice in the

Acquisition of Expert Performance.”

13 Ibid.; and Singer et al., Discipline-Based Education Research.
14 Schwartz et al., The ABCs of How We Learn.
15 Here we are considering the brain of the typical university student, neglecting any

“clinical” anomalies present in special cases.

16 There are many calls for university students to learn “critical thinking.” As this is
usually defined, it is equivalent to making better decisions in realistic situations.
But a closer examination of what this means to any particular advocate of teach-
ing critical thinking is usually that the students should learn to use the skills and
knowledge of their discipline in making decisions of the sort valued by their dis-
cipline, with the assumption that this represents a generic skill that all students
should have. There is an extensive body of research indicating that there is no such
generic skill: any authentic decisions will necessarily involve discipline-specific
knowledge and reasoning, and hence any measure of “critical thinking,” including
those currently used with claims they are generic, such as the Collegiate Learning
Assessment, are in fact not generic. If the context and nature of the decisions in-
volved changed, so would a student’s performance.

17 Schwartz et al., The ABCs of How We Learn.
18 Ibid.
19 This is different from the all-too-common example of a novice teacher applying
some technique without understanding the principles on which it is based or the
benefits it might provide, and thereby achieving little apparent benefit. An exam-
ple (a real one) is introducing the use of clicker questions and peer discussion in a
political science course with little understanding of suitable questions or goals of

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

discussion, and then judging the effectiveness of this teaching method according
to the changes (or not) observed in the quality of the writing of the students’ term
papers.

20 Freeman et al., “Active Learning Increases Student Performance in Science, Engi-

neering, and Mathematics.”

21 Carl Wieman, “A Better Way to Evaluate Undergraduate Teaching,” Change: IL
Magazine of Higher Learning 47 (1) (2015): 6–15; Lillian MacNell, Adam Driscoll, E
Andrea N. Hunt, “What’s in a Name: Exposing Gender Bias in Student Ratings of
Teaching,” Innovative Higher Education 40 (4) (2015): 291–303; and Amy L. Graves,
Estuko Hoshino-Browne, and Kristine P. H. Lui, “Swimming against the Tide:
Gender Bias in the Physics Classroom,” Journal of Women and Minorities in Science and
Engineering 23 (1) (2017).

22 Wieman, “A Better Way to Evaluate Undergraduate Teaching.”
23 Originally developed for characterizing teaching in sciences, with some very small
wording changes, it is now being used on at least a limited basis for all academic
disciplines. Carl Wieman and Sarah Gilbert, “The Teaching Practices Inventory:
A New Tool for Characterizing College and University Teaching in Mathematics
and Science,” CBE–Life Sciences Education 13 (3) (2014): 552–569; and Carl Wieman
Science Education Initiative at the University of British Columbia, “CWSEI Teach-
ing Practices Inventory,"Ottobre 3, 2014, http://www.cwsei.ubc.ca/Files/CWSEI
_TeachingPracticesInventory_Oct2014.pdf.

24 Carl Wieman, Improving How Universities Teach Science: Lessons from the Science Education

Initiative (Cambridge, Massa.: Stampa dell'Università di Harvard, 2017).

25 Ibid.
26 Ibid.
27 Ibid.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

Appendix I
CWSEI Teaching Practices Inventory:
For Use in the Natural and Social Sciences

To create the inventory we devised a list of the various types of teaching prac-
tices that are commonly mentioned in the literature. We recognize that these
practices are not applicable to every course, and any particular course would
likely use only a subset of these practices.

We have added places that you can make additions and comments and we
welcome your feedback.

It should take only about 10 minutes to fill out this inventory.

Please fill out the inventory for the current or just completed Term, lecture
sections only.

Course number:
Sezione #(S) or Instructor name:
Total number of students in your class or section (approximate):

IO. Course information provided to students via hard copy or course
webpage

Check all that occurred in your course:

 List of topics to be covered
 List of topic-specific competencies (skills, expertise, . . .) students

should achieve (what students should be able to do)

 List of competencies that are not topic related (critical thinking,

problem solving, . . .)

 Affective goals–changing students’ attitudes and beliefs (interesse,
motivation, pertinenza, beliefs about their competencies, how to
master the material)
 Other (please specify)

If you selected other, please specify:

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

II. Supporting materials provided to students

Check all that occurred in your course:

 Student wikis or discussion boards with little or no contribution

from you

 Student wikis or discussion boards with significant contribution

from you or TA

 Solutions to homework assignments
 Worked examples (testo, pencast, or other format)
 Practice or previous year’s exams
 Animations, video clips, or simulations related to course material
 Lecture notes or course PowerPoint presentations (partial/skeletal

or complete)

 Other instructor selected notes or supporting materials, pencasts,

eccetera.

 Articles from related academic literature
 Examples of exemplary papers or projects
 Grading rubrics for papers or large projects
 Other (please specify)

If you selected other, please specify:

III. In-class features and activities

UN. Various

Give approximate average number:

Average number of times per class: pause to ask for questions:
Average number of times per class: have small group discussions or
problem solving:
Average number of times per class: show demonstrations, simulations,
or video clips:
Average number of times per class: show demonstrations, simulations,
or video where students first record predictions (write down, eccetera.) E
then afterwards explicitly compare observations with predictions:
Average number of discussions per term on why material useful and/or
interesting from students’ perspective:
Comments on above (if any):

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

Check all that occurred in your course:

 Students asked to read/view material on upcoming class session
 Students read/view material on upcoming class session and com-
plete assignments or quizzes on it shortly before class or at begin-
ning of class

 Reflective activity at end of class, e.g. “one-minute paper” or simi-
lar (students briefly answering questions, reflecting on lecture and/
or their learning, eccetera.)

 Student presentations (verbal or poster)

Fraction of typical class period you spend lecturing/talking to whole class
(presenting content, deriving mathematical results, presenting a problem
solution, . . .):

 0–20%
 20–40%
 40–60%
 60–80%
 80–100%

Considering the time spent on the major topics, approximately what frac-
tion was spent on the process by which the theory/model/concept was de-
veloped, including the experimental methods and results that support spe-
cific theories?

 0–10%
 11–25%
 more than 25%

B. Individual Student Responses (ISR)

If a student response method is used to collect responses from all students
in real time in class, what method is used?

Check all that occurred in your course:

 Raising hands
 Raising colored cards
 Electronic (e.g. “clickers”) with student identifier

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

 Electronic anonymous
 Written student responses that are collected and reviewed in real time
 Other (please specify)

If you selected other, please specify:

Number of ISR questions posed followed by student-student discus-
sion per class:
Number of times ISR used as quiz (counts for marks and no student dis-
cussion) per class:

IV. Assignments

Check all that occurred in your course:

 Homework/problem sets assigned or suggested but did not contrib-

ute to course grade

 Homework/problem sets assigned and contributed to course grade

at intervals of 2 weeks or less

 Paper or project (an assignment taking longer than two weeks and in-
volving some degree of student control in choice of topic or design)
 Encouragement and facilitation for students to work collaboratively

on their assignments

 Explicit group assignments
 Other (please specify)

If you selected other, please specify:

V. Feedback and testing; including grading policies

UN. Feedback from students to instructor during the term

Check all that occurred in your course:

 Midterm course evaluation
 Repeated online or paper feedback or via some other collection

means such as clickers
 Other (please specify)

If you selected other, please specify:

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

B. Feedback to students

(check all that occurred in your course)

 Assignments with feedback from instructor, teaching assistant, O
peer before grading or with opportunity to redo work to improve
grade

 Students see graded assignments
 Students see assignment answer key and/or grading rubric
 Students see graded midterm exam(S)/quizzes
 Students see midterm exam(S)/quizzes answer key(S)
 Students explicitly encouraged to meet individually with you
 Other (please specify)

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

If you selected other, please specify:

C. Testing and grading

Number of tests during term that reflect course expectations (e.g. mid-
term exams, but not final exams):

Approximate fraction of test scores from questions that required students
to explain reasoning:

Approximate breakdown of course grade (% in each of the following
categorie):

Final exam:
Midterm/other exam(S):
Homework assignments:
Paper(S) or project(S):
In-class activities:
In-class quizzes:
Online quizzes:
Participation:
Lab component:
Other:

If you selected other, please specify:

148 (4) Fall 2019Carl Edwin Wieman

VI. Other

Check all that occurred in your course:

 Assessment given at beginning of course to assess background

knowledge

 Use of instructor-independent pre-post test (e.g. as concept inven-

tory) to measure learning

 Use of a consistent measure of learning that is repeated in multiple

offerings of the course to compare learning

 Use of pre-post survey of student interest and/or perceptions about

the subject

 Opportunities for students’ self-evaluation of learning
 Students provided with opportunities to have some control over
their learning, such as choice of topics for course, paper, or project,
choice of assessment methods, eccetera.

 New teaching methods or materials were tried along with measure-

ments to determine their impact on student learning

VII. Training and guidance of Teaching Assistants

Check all that occurred in your course:

 No TAs for course
 TAs must satisfy English language skills criteria
 TAs receive 1/2 day or more of training in teaching
 There are Instructor-TA meetings every two weeks or more fre-
quently where student learning and difficulties and the teaching of
upcoming material are discussed

 TAs are undergraduates
 TAs are graduate students
 Other (please specify)

If you selected other, please specify:

VIII. Collaboration or sharing in teaching

 Used or adapted materials provided by colleague(S)
 Used “Departmental” course materials that all instructors of this

course are expected to use

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

Discussed how to teach the course with colleague(S):

 1 Never
 2
 3
 4
 5 Very Frequently

Read literature about teaching and learning relevant to this course:

 1 Never
 2
 3
 4
 5 Very Frequently

Sat in on colleague’s class (any class) to get/share ideas for teaching:

 1 Never
 2
 3
 4
 5 Very Frequently

IX. General

Open-ended comments:

Please write any other comments here. If this inventory has not captured
an important aspect of your teaching of this course, or you feel you need to
explain any of your above answers, please describe it here:

Approximately how long did it take you to fill out this inventory?

We thank you for taking the time to fill out this inventory.

Fonte: Adapted from Carl Wieman and Sarah Gilbert, “Teaching Practices Inventory,"
CBE–Life Sciences Education 13 (3) (2014): 552–569.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

Appendix II
Background of the CWSEI

The Carl Wieman Science Education Initiative (CWSEI) at the University of
British Columbia and its smaller partner at the University of Colorado Boul-
der were large-scale finite-duration experiments (approximately $10 million and $5 million, rispettivamente) in institutional change. They showed that it is
possible for large research-intensive university science departments to make
major changes in their teaching, and they revealed the processes that help and
hinder such change. An extensive discussion of this experiment is given in
Carl Wieman, Improving How Universities Teach Science (2017).

At the University of British Columbia, the Initiative changed the teaching
of about 170 science faculty members and courses, with the fraction of trans-
formed faculty and credit hours reaching 90 percent in some departments.
These faculty are finding teaching to be more rewarding, and their students
are far more engaged and learning more. Teaching became much more of a
collaborative intellectual activity in these departments, with faculty shar-
ing methods and results and seeking out ideas from others. The transformed
teaching is characterized by: detailed learning goals for the course that express
what students should learn to do in operational terms; in-class active-learning
activities such as peer instruction, think-pair-share, and worksheets that have
students practicing expert thinking by answering questions in small groups
monitored by the instructor and TAs and interspersed with regular instruc-
tor feedback and guidance; different forms of assessment aligned with course
goals, such as graded homework, more-frequent lower-stakes exams, E
two-stage exams that students complete individually and then as a group; Rif-
flective exercises such as two-minute papers at the end of a class; and brief
preclass preparations such as targeted readings.

Such results were not easy nor shared across all departments. The three
most important elements were: supporting department-level change, incen-
tives, and maximizing faculty buy-in.

Supporting department-level change. At universities, each department de-
cides what and how to teach, and so the department is the unit of educational
change. The CWSEI used a competitive grant program by which departments
competed for up to $1.8 million over six years to transform teaching. Potential
grants of this scale produced discussions of undergraduate teaching needs and
opportunities that had never happened before. The success of the funded de-
partments was strongly influenced by disciplinary culture and the quality of
the departmental leadership and administration, which varied greatly. Nuovo

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching

structures and people, such as a teaching initiatives committee with respon-
sibility and resources, were required, as the traditional departmental struc-
tures, when left unchanged, were never effective at supporting innovation.

A key component in every successful department were science education
specialists (SESs) with deep expertise in the respective discipline combined
with expertise in teaching and learning in the discipline. The SESs were hired
by the department and worked collaboratively with a sequence of faculty to
transform courses and, in the process, the teaching of the faculty. The SESs act
as nonthreatening coaches, providing expert guidance and support to facul-
ty members as they try new things in their courses. With SES guidance, a fac-
ulty member was likely to implement research-based teaching methods in an
effective manner from the beginning, and hence have a positive teaching ex-
perience. The SESs also provide expert and time-saving assistance in devel-
oping new course materials and assessments. It was usually easy to find good
SES candidates with the necessary disciplinary knowledge and interest in ed-
ucation, typically new Ph.D.s, but it was necessary to set up an extensive train-
ing program for them in the relevant research and best research-based teach-
ing methods.

Incentives. Incentives need to be provided for both the departments and the
individual faculty members to take the time to learn new teaching methods.
The formal incentive system is a powerful disincentive to improving teach-
ing. At all universities, the evaluation system does not recognize that research
has shown there are fundamental differences in the effectiveness of different
teaching methods, and hence the system penalizes any time away from re-
search to learn better methods. The CWSEI showed that it does not cost more
money or time to teach using these more effective methods, but it does cost
money to bring about change. One incentive is having the dean and depart-
ment chair clearly convey that better teaching is an important institutional
goal, but most other incentives involve money in one form or another, largely
to minimize and compensate for the time required to learn.

Maximizing faculty buy-in. Instead of starting with specific courses to trans-
form, it was more effective to start with any willing faculty members and ac-
commodate them according to what courses and process of change work best
for them. Some faculty were happy to carry out a total course transformation
all at once, but for many others, an incremental approach worked better, from
both psychological and logistical perspectives. Even modest changes usually
showed positive results. Almost immediately the use of active learning meth-
ods gave faculty a better understanding of their students’ thinking, and hence
how to make their teaching more effective. There are many fears associated
with making change. The most effective ways to address these fears were not

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

148 (4) Fall 2019Carl Edwin Wieman

by providing data, but rather by having faculty talk to their colleagues who
had transformed their teaching and watch the teaching of a good transformed
course in their department. For many faculty members, it can take one or two
years of hearing about these ideas and discussing them with their colleagues
before they decide to change, with no obvious large differences between
young and older faculty members.

The CWSEI has published a large body of resources on its website. These in-
clude peer-reviewed research papers on various aspects of teaching and learn-
ing and extensive guidance for instructors. The following links also feature a
variety of guides on details of design and implementation of research-based
instruction and videos showing demonstrations.

• For a collection of documents offering detailed advice for departments
and faculty members on how to redesign courses, see “Course Trans-
formation Resources,” http://www.cwsei.ubc.ca/resources/course_
transformation.htm.

• For a collection of short guides for instructors (on assessment, clicker
use, student engagement, and so on) that illustrates in concrete terms
the pedagogical philosophy (active engagement of students) underly-
ing these initiatives, see “Instructor Guide,” http://www.cwsei.ubc
.ca/resources/instructor_guidance.htm. The advice is highly practical.
• For a collection of videos that show, among other things, what active
learning looks like, see “Science Education Initiative (SEI) Videos,"
http://www.cwsei.ubc.ca/resources/SEI_video.html.

• For an annotated bibliography of papers on the research behind many
aspects of active learning, see “Recommended Papers,” http://www
.cwsei.ubc.ca/resources/papers.htm.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
D
UN
e
D
UN
R
T
io
C
e
–
P
D

F
/

1
4
8
4
4
7
1
8
3
1
3
5
9
D
UN
e
D
_
UN
_
0
1
7
6
0
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Dædalus, the Journal of the American Academy of Arts & SciencesExpertise in University Teaching
Scarica il pdf