Methods - Ricerca sull'intelligenza artificiale specializzata al MIT

Methods

Bridging global and local topology in whole-brain
networks using the network statistic jackknife

Teague R. Henry

1, Kelly A. Duffy1, Marc D. Rudolph1, Mary Beth Nebel2,3,

Stewart H. Mostofsky2,3,4, and Jessica R. Cohen1

1Department of Psychology and Neuroscience, University of North Carolina at Chapel Hill, Chapel Hill, NC, USA
2Center for Neurodevelopmental and Imaging Research, Kennedy Krieger Institute, Baltimore, MD, USA
3Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD, USA
4Department of Psychiatry and Behavioral Science, Johns Hopkins School of Medicine, Baltimore, MD, USA

Keywords: Network, Statistics, fMRI, Functional connectivity, Graph theory, Whole-brain
analysis, Jackknife

a n o p e n a c c e s s

j o u r n a l

ABSTRACT

Whole-brain network analysis is commonly used to investigate the topology of the brain
using a variety of neuroimaging modalities. This approach is notable for its applicability to a
large number of domains, such as understanding how brain network organization relates to
cognition and behavior and examining disrupted brain network organization in disease. UN
beneﬁt to this approach is the ability to summarize overall brain network organization with a
single metric (per esempio., global efﬁciency). Tuttavia, important local differences in network
structure might exist without any corresponding observable differences in global topology,
making a whole-brain analysis strategy unlikely to detect relevant local ﬁndings. Conversely,
using local network metrics can identify local differences, but are not directly informative of
differences in global topology. Here, we propose the network statistic (NS) jackknife
framework, a simulated lesioning method that combines the utility of global network analysis
strategies with the ability to detect relevant local differences in network structure. Noi
evaluate the NS jackknife framework with a simulation study and an empirical example
comparing global efﬁciency in children with attention-deﬁcit/hyperactivity disorder (ADHD)
and typically developing (TD) children. The NS jackknife framework has been implemented
in a public, open-source R package, netjack, available at
https://cran.r-project.org/package=netjack.

INTRODUCTION

Describing the brain as a network, an approach termed network neuroscience (Bassett &
Sporns, 2017), is a powerful method that allows regions distributed across the entire brain
to be incorporated into a single model describing overall brain topology. This method has pro-
vided insights into whole-brain disruption in neurological and psychiatric diseases (Fornito,
Bullmore, & Zalesky, 2017; Fornito, Zalesky, & Breakspear, 2015), the dynamic processes
underlying cognition (Cohen & D’Esposito, 2016; Kucyi, Tambini, Sadaghiani, Keilholz, &
Cohen, 2018), and changes across development (Baum et al., 2017; Grayson & Fair, 2017).
Researchers using a network neuroscience framework can draw upon the rich methodological
and statistical literature in graph theory and network analysis, grounding the approach in a
well-established mathematical framework.

Tuttavia, whole-brain analysis methods, as commonly applied in network neuroscience,
often fail at what Hallquist and Hillary (Hallquist & Hillary, 2018) refer to as “matching theory

Citation: Henry, T. R., Duffy, K. A.,
Rudolph, M. D., Nebel, M. B.,
Mostofsky, S. H., & Cohen, J. R. (2020).
Bridging global and local topology in
whole-brain networks using the
network statistic jackknife. Network
Neuroscience, 4(1), 70–88. https://
doi.org/10.1162/netn_a_00109

DOI:
https://doi.org/10.1162/netn_a_00109

Supporting Information:
https://doi.org/10.1162/netn_a_00109
https://CRAN.R-project.org/
package=netjack

Received: 4 Giugno 2019
Accepted: 12 settembre 2019

Competing Interests: The authors have
declared that no competing interests
exist.

Corresponding Author:
Teague Rhine Henry
trhenry@email.unc.edu

Handling Editor:
Richard Betzel

Copyright: © 2019
Istituto di Tecnologia del Massachussetts
Pubblicato sotto Creative Commons
Attribuzione 4.0 Internazionale
(CC BY 4.0) licenza

The MIT Press

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Jackknife:
A nonparametric statistical technique
that involves repeatedly reestimating
a parameter value after the removal
of single observations.

Whole-brain (global)
network statistic:
A statistic derived from a whole-brain
network that describes an aspect of
global topology.

Global topology:
The overall shape and structure of a
rete.

to scale.” This refers to the fact that whole-brain network analysis methods operate at a global
resolution, while relevant differences might be very localized. In other words, there may be
differences at a local scale (cioè., within a single subnetwork) that would be missed when cal-
culating a single, whole-brain summary measure that describes global topology. Conversely,
observed global differences in topology could be driven by a speciﬁc local difference (cioè., an
observed overall difference in network integration could be driven by a difference limited to
a single subnetwork). This mismatch in resolution, Perciò, could obscure meaningful differ-
ences in network structure. While local network statistics, such as participation coefﬁcient of
a single node, can be used to identify local differences of interest, these local statistics do not
directly assess a localized difference in global topology, and as such are not ideal for matching
a global theory to a local scale. In the present manuscript, we develop the network statistic
(NS) jackknife as a general method to localize global network statistics across a variety of
network metrics and brain features of interest, thus allowing researchers to better address this
resolution issue.

The NS jackknife method (available for use at https://cran.r-project.org/package=netjack)
proposed here allows researchers to identify speciﬁc brain regions, connections, or subnet-
works (cioè., local features) that drive global network analysis results or, conversely, that are not
detected by global network analysis. By doing so, our method can detect local differences in
network structure using the same summary metrics that a global analysis would use, as op-
posed to relying on local metrics that do not explicitly link to the global metrics in question.
This approach bridges the gap between a global and a local scope of analysis by allowing
researchers to apply global analysis strategies in a way that reveals differences at local scopes,
even when those differences would not be detected with the global analysis method. We eval-
uate the performance of the NS jackknife with a simulation study and demonstrate the utility
of this method with an empirical example comparing resting-state brain network organization
in children with attention-deﬁcit/hyperactivity disorder (ADHD) and age-matched typically
developing (TD) children.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Global Versus Local Topology

There are many network statistics that can be used to analyze the global topology of a network;
for simplicity we focus on two widely used network statistics as examples. Primo, we use mod-
ularity, or a measure of a network’s segregation into multiple communities (or subnetworks;
Newman, 2006). Secondo, we use global efﬁciency, a statistic that assesses the overall inte-
gration of a network (Latora & Marchiori, 2001). These two statistics have led to great insight
regarding brain organization, in particular for supporting the hypothesis that the balance be-
tween network segregation (dense, within-subnetwork connectivity) and network integration
(communication across distinct subnetworks) is critical for cognition (for reviews and canon-
ical articles, see the following: Cohen & D’Esposito, 2016; Shine & Poldrack, 2017; Sporns,
2013; Sporns, Chialvo, Kaiser, & Hilgetag, 2004).

Tuttavia, a key issue for whole-brain analysis is its poor detection of local differences in
topology. Per esempio, if a whole-brain analysis produces differences in overall network topol-
ogy between groups or conditions, it does not provide insight into where in the brain these
differences might arise. Further, if a whole-brain analysis produces no differences in overall
network topology, that does not preclude the existence of important local differences. Questo
property of whole-brain network analysis is less than ideal as it ignores local features, Quale
can include speciﬁc brain regions, speciﬁc connections between brain regions, or speciﬁc
subnetworks. A local topological difference arising from one region/connection/subnetwork

Network Neuroscience

The network statistic jackknife

Speciﬁcity:
When differences in global topology
are being driven by speciﬁc
differences in local topology, COME
opposed to globally distributed
differences.

Equiﬁnality:
When different network
conﬁgurations can lead to the same
global network statistic.

versus another could have vastly different implications for empirical inferences and resultant
clinical applications.

There are several existing graph theory metrics that operate on a local scale. For exam-
ple, within-module degree indicates which nodes are more central to their communities, E
participation coefﬁcient indicates which nodes connect different communities (Guimerà &
Nunes Amaral, 2005). Additionally, there are a variety of statistical models for identifying
nodes and other network features of importance. One of the most widely used is the network-
based statistic method (NBS; Zalesky, Fornito, & Bullmore, 2010), which identiﬁes differences
between groups by examining maximally different sets of edges. The degree-based statistic
method of Yoo and colleagues (Yoo et al., 2017) is more focused than that of the NBS method,
and identiﬁes highly central nodes that are related to a hypothesis of interest. Another recent
approach, that of screen ﬁltering (Meskaldji et al., 2015), allows researchers to analyze group
differences in nodal and edge-wise metrics; Tuttavia, this method as currently implemented
is restricted to nodal, edge-wise or subnetwork statistics as it relies on correcting a collection
of statistical tests. All of these methods allow researchers to assess local differences between
groups and thus are powerful tools for understanding different features of network organi-
zation, but none of these methods assess local drivers of global topological properties of a
rete. Per esempio, a common method for examining group differences in local modular
structure is examining differences in participation coefﬁcient for individual nodes or subsets
of nodes. While nodes that exhibit a participation coefﬁcient difference might correspond to
nodes that are relevant to differences in global modularity, this is by no means assured. Local
metrics and tests can be related to global metrics (cioè., participation coefﬁcient can be related
to whole-brain modularity), but these relations are not one-to-one, and a difference in local
properties does not necessarily imply a difference in global properties. Allo stesso modo, previous sta-
tistical models that identify network elements of importance do not identify these elements
based on their relative contribution to global topology, but rather based on their impacts on
local topology. These properties of local metrics and existing local statistical models suggest
a need for a framework that can assess differences in global topology and directly link these
global differences to local structures.

The disconnect that arises between the global and the local scope can be due to speciﬁcity,
in which whole-brain network results are driven by speciﬁc connections, regions, or subnet-
works; or to equiﬁnality, in which strikingly different network conﬁgurations can lead to the
same whole-brain summary metric values. These two concepts can be related, as equiﬁnality
can be caused by speciﬁc differences in local topology that do not lead to differences in sum-
mary metrics describing global topology. The NS jackknife, proposed here, provides a solution
to these issues. This statistical approach allows researchers to simultaneously analyze differ-
ences in global topology and localize these differences to speciﬁc brain regions, connections,
or subnetworks.

Network Speciﬁcity and Equiﬁnality

Speciﬁcity is the network principle whereby differences in observed whole-brain or whole-
network topology can be driven by speciﬁc differences in local topology, as opposed to glob-
ally distributed differences. A whole-brain network difference can be speciﬁc if, Per esempio,
one group of interest has greater connectivity within one speciﬁc functional subnetwork than
a different group, and this increase in connectivity is substantial enough that it alone drives a
signiﬁcant difference in global modularity. Conversely, a whole-brain network difference can
be nonspeciﬁc if the difference in topology is due to a diffuse pattern of local differences.

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Per esempio, if one group has higher connectivity between all pairs of subnetworks, this can
result in a difference in global efﬁciency that cannot be localized to any given subset of net-
works. One key issue with whole-brain network analysis is that it is not capable of determining
whether an observed topological difference is speciﬁc or nonspeciﬁc. This property has been
termed the theory-to-scale issue (Hallquist & Hillary, 2018).

This lack of speciﬁcity is complicated by the fact that, when calculating various summary
statistics for brain networks, typical approaches in network neuroscience can produce the
same network statistics for different network conﬁgurations. Equiﬁnality, a term borrowed from
system theory (Bertalanffy, 1950), is the network principle whereby the same result can be
achieved by many different means. Here, we use it to emphasize the fact that the same in-
ference can result from different global network conﬁgurations. Per esempio, Figura 1 shows
two networks, each consisting of two subnetworks. The difference between these networks is
in how these subnetworks are connected. In Network A, the communities are connected via a
hub node (dotted circle), while for Network B the subnetworks are connected more diffusely.
Whole-network modularity of the two networks in Figure 1 is identical, yet this result masks
the fact that the local structure of the inter-subnetwork connections in the two networks is quite
different. This demonstrates the possible disconnect between whole-brain ﬁndings (cioè., global
scope) and speciﬁc local differences in topology (cioè., local scope).

Speciﬁc versus nonspeciﬁc whole-brain ﬁndings occur with regularity in neuroimaging re-
search. Per esempio, individuals suffering from Alzheimer’s disease exhibit reductions in de-
fault mode network connectivity as well as global loss of a small-world network structure
(Pievani, de Haan, Wu, Seeley, & Frisoni, 2011). This analysis of whole-brain small-world
structure differentiates patients with Alzheimer’s disease from healthy control participants but
provides no information as to the role default mode network connectivity plays in that global
ﬁnding. The NS jackknife method has the ability to quantify whether reductions of whole-brain
small-world structure in Alzheimer’s disease are driven by the default mode network alone, are
widely distributed across all subnetworks in the brain, or are driven by a small number of sub-
networks.

Figura 1. Two networks with the same value of modularity yet very different inter-subnetwork
patterns of connectivity. Note that Network A contains a single connector hub, or a node that links
two communities (dashed circle), while Network B does not.

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Virtual/simulated lesioning:
A technique in neuroimaging that
removes components of a network
during analysis to examine the
impact of those components on
global topology.

Network robustness:
How much a network’s global
topology changes when nodes/edges
are removed. This removal is
typically done in a random or
semirandom fashion to assess the
network’s robustness to attack or
damage.

These speciﬁc local differences with prominent global differences found in Alzheimer’s
disease can be contrasted with the ﬁnding that individuals with autism spectrum disorder
(ASD) tend to exhibit increased whole-brain integration, as assessed with global efﬁciency,
as compared with healthy control participants (Lewis, Theilmann, Townsend, & Evans, 2013;
Rudie & Dapretto, 2013). While other research has shown speciﬁc differences between indi-
viduals with ASD and healthy control participants, for example within the salience network
(Di Martino et al., 2009), the ﬁnding of increased functional integration appears be a result of
more diffuse and weaker functional connections distributed across the entire brain, and thus a
whole-brain phenomenon (Roine et al., 2015; Rudie & Dapretto, 2013). While a whole-brain
network analysis would accurately detect this group difference, it would give no indication
whether the difference was due to a widely distributed whole-brain difference or driven by
speciﬁc subnetworks or connections, a conclusion that could be made with use of the NS jack-
knife. These examples emphasize that whole-brain network analysis strategies are designed to
examine a global scope but lack the ability to determine the speciﬁcity of any given ﬁnding
(Hallquist & Hillary, 2018).

Localization of Whole-Brain Analysis Using the NS Jackknife

The NS jackknife localizes whole-brain network analyses by iteratively removing elements of
the network, such as edges, nodes, or subnetworks, and recalculating the target network statis-
tic without those elements. By sweeping through the whole network, as well as assessing the
impact of the removal across subjects, the NS jackknife creates an empirical distribution of
network element effects, allowing a researcher to localize speciﬁc features of brain network
organization that contribute most strongly to a given whole-brain network statistic. The NS
jackknife was inspired by the traditional jackknife estimator used extensively in nonparametric
statistics and can be considered a virtual lesioning technique. We are using the term “jackknife”
not only because the NS jackknife relies on the leave-one-out approach of the traditional jack-
knife, but also because like the traditional jackknife, the NS jackknife is a general statistical
technique that can be applied to any question that entails localizing a global network statistic.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

A jackknife in statistics is a resampling technique used to adjust for outliers using a leave-
one-out approach (Efron, 1982; Tukey, 1958). This procedure removes a single observation,
such as a single subject’s data, from a sample, recalculates the effect estimate, then adds the
removed observation back into the sample and removes another; it iterates through the sample
in this fashion. In standard sampling situations, in which each observation is the behavioral data
of a single individual, this method is useful for detecting potential outliers. Jackknife techniques
have been previously used in network science to assess the overall robustness of a network
statistic to node removal (Snijders & Borgatti, 1999). Tuttavia, previous uses of this technique
have focused on single instances of a network (cioè., one social network) and on obtaining
estimates of a network statistic, rather than on assessing the importance of particular network
structures in a set of multiple networks (cioè., multiple brain networks).

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

While a standard jackknife procedure removes individual subjects in turn from an analysis,
we are proposing here to remove elements of a graph, such as nodes, edges, or subnetworks.
Here for the sake of example, we focus on the removal of entire functional subnetworks, O
tightly interconnected sets of regions of interest (ROIs) thought to underlie various aspects of
cognition, such as the default mode network (Greicius, Krasnow, Reiss, & Menon, 2003) or the
salience network (Seeley et al., 2007). Once a subnetwork is removed, the network statistic is
recalculated, and the procedure continues with the removal of another subnetwork. Speciﬁc
versions of this procedure have been used in previous neuroimaging research (per esempio., Achard,

Network Neuroscience

The network statistic jackknife

Salvador, Whitcher, Suckling, & Bullmore, 2006; Henry, Dichter, & Gates, 2018); we develop
a general framework here. Critically, while our NS jackknife estimator is based on previous
work in simulated lesioning, and at its core is a simulated lesioning method (Achard et al.,
2006; Alstott, Breakspear, Hagmann, Cammoun, & Sporns, 2009; Crucitti, Latora, Marchiori,
& Rapisarda, 2004), it is focused on localization of network effects rather than the overall
robustness of a network. More speciﬁcally, while robustness focuses on understanding the
overall resiliency of a network to attack, localization attempts to understand the impact of the
removal of speciﬁc elements on the network. Robustness and localization are highly related,
and the NS jackknife differs from previous simulated lesioning methods mainly in interpretation
and in its speciﬁc focus on understanding group differences. Additionally, while the focus here
is on testing group differences without subject- or network-level covariates, the NS jackknife
framework can be extended to more complex models that can account for covariates and
other design factors. In this sense, the NS jackknife is an extension of the simulated lesioning
technique into a general testing framework for localizing global network statistics. Finalmente, we
provide an easily accessible implementation of the NS jackknife in the netjack package.

The NS Jackknife

To describe the NS jackknife in more precise terms, let {G1, . . . Gn} be the observed networks
of n subjects. Let f (·) be the functional form of some network statistic, such as modularity,
and let k be the indicator of a feature of interest (such as a subnetwork), taken from some set
K of size |K|. Gi[−k] is the network of individual i with feature k removed. More concretely, if
we were analyzing functional connectivity networks, then Gi would be the whole-brain func-
tional connectivity network of individual i, and Gi[−DMN] would be the functional connectivity
network of individual i with all the nodes of the default mode network removed.

The set of original network statistics is f (·) applied to all G1, . . . Gn, and this n-length vector
is denoted as ˆf . The jackknife estimate with regard to feature k is then f (·) applied to all
G1[−k], . . . , Gn[−k]; this length n vector is denoted as ˆf−k. Finalmente, the difference between the
original network statistic and the jackknife estimator with regard to structure k is denoted as
ˆd−k.

Applying this procedure to the |K| features of interest (per esempio., all functional subnetworks) leads
A |K| jackknife estimate vectors and |K| difference vectors for each subject. If subjects are
(B)
divided between Groups A and B, we denote the estimate and difference vectors as ˆf
−k
and ˆd

(UN)
−k

, ˆf

(UN)

−k , ˆd

(B)
−k .

There are several methodological considerations when using the NS jackknife. The ﬁrst is
that network statistics are a priori dependent on network size and density (van Wijk, Stam,
& Daffertshofer, 2010); comparing network statistics across different subsets of networks of
different sizes can lead to spurious ﬁndings. In the NS jackknife framework, this issue is handled
by comparing networks of the same size (as in the group test) or changes in network statistics
(as in the differential impact test).

Additionally, the NS jackknife produces a series of dependent multiple comparisons. In tra-
ditional jackknife settings, observations removed at each iteration are independent of all other
observations. Tuttavia, in the NS jackknife the removal of individual elements from every sub-
ject’s network and the subsequent reestimation of network statistics results in a series of depen-
dent statistical tests. Because of the dependent nature of these tests, and the risk for inﬂation
because of multiple comparisons, the Benjamani-Hochberg (BH) procedure is an appropriate
false discovery rate (FDR) correction for multiple comparisons (Benjamini & Hochberg, 1995).

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Other multiple comparisons corrections that allow for dependent tests, such as the Benjamani-
Yekutieli (BY; Benjamini & Yekutieli, 2001) correction, are also appropriate, while the Bonfer-
roni correction would not be, as it does not account for the presence of dependent tests. If the
network components of interest are nodes or edges, and the subnetwork membership of these
components is known, the screen ﬁltering approach of Meskaldji et al. (2015) provides a way
of correcting for multiple comparisons that results in more power to detect effects than the BH
or BY corrections, and should be preferentially used.

All analyses and graphical outputs in this manuscript use the netjack open source R pack-
age (Henry, 2018), a package tailor-made for performing the NS jackknife in a simple to use,
customizable fashion.

Localizing Group Differences and Differential Impact

The primary use of the NS jackknife is to localize global network statistic differences be-
tween groups to speciﬁc subnetworks, nodes, or edges. Consider the example of comparing
the whole-brain modularity for a given set of subnetworks between two clinical groups. Nell'annuncio-
dition to quantifying differences across the groups in global modularity using standard network
statistics, the NS jackknife allows for two additional questions of substantive interest:

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

1. Group differences: Given the removal of a subnetwork, are the two groups’ mean global
values signiﬁcantly different from each other? This corresponds to a hypothesis of the
following:

(UN)
ˆf
−k i 6= E h
2. Differential impact: Is there a differential impact of the removal of a subnetwork between

(B)
ˆf
−k i .

E h

the two groups? This corresponds to a hypothesis of the following:

E h

(UN)
ˆd
−k i 6= E h

(B)
ˆd
−k i .

We illustrate the use of these two tests in the following simulated example.

Data generation. For Group A, the sample of networks was generated using a stochastic
block model for binary undirected networks (Nowicki & Snijders, 2001), with probabilities set
to create the observed community structure (0.9 within, 0.5 between). The sample contained
10 “subjects.” For Group B, data generation used the same procedure and the same base prob-
abilities (0.9 within, 0.5 between), with additional connections between the yellow and black
groups set to 0.5 probability (Guarda la figura 2).Group B, like Group A, contained 10 “subjects.”

Analysis. The NS jackknife was used to quantify the effect of the removal of each of the ﬁve
subnetworks on each subject’s overall network modularity. For the group differences analysis,
a two-sample t test was conducted to determine whether the updated modularity due to the
removal of any given subnetwork was signiﬁcantly different across groups. For the differential
impact analysis, a two-sample t test was conducted on the difference in modularity due to the
removal of any given subnetwork to determine whether there was a differential impact across
the groups in removing each subnetwork. For the differential impact analysis, p values were
corrected for ﬁve tests using the BH correction for dependent multiple comparisons. Group
difference test p values are uncorrected.

Results. Whole-brain modularity was signiﬁcantly higher in Group B than in Group A
(T(13.41) = 2.9288, P < 0.05; average modularity 0.1974 and 0.1332, respectively). For the group differences analysis, the whole-brain group difference (B > UN) remained after removal of

Network Neuroscience

The network statistic jackknife

Figura 2. Two modular networks. The red module acts as a central hub for both networks, while
in Network A the black and yellow modules are directly interconnected.

the blue, green, or red subnetworks, while removal of the black or yellow subnetworks resulted
in no group difference in modularity (Figura 3, left panel). For the differential impact analysis,
removal of the black and yellow subnetworks resulted in an increased change in modularity for
Group A as compared with Group B. Removal of the red subnetwork resulted in an increased
change in modularity for Group B as compared with Group A (although modularity increased

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Figura 3. Group differences (left ﬁgure) and differential impact (right ﬁgure) of subnetwork removal
on modularity of Network A (in blue) and Network B (in green) from Figure 2. Central line is the
median, box is the interquartile range (IQR), and whiskers represent 1.5 ∗ IQR. Points represent
values greater than 2 SD from the mean. Both group difference and differential impact tests suggest
that the red network is signiﬁcantly more centrally integrated in Group B than in Group A. IL
differential impact tests additionally show that removal of the black or yellow network causes more
of an increase in modularity for Group A than Group B, suggesting that these subnetworks are more
integrated into the whole network in Group A.

Network Neuroscience

The network statistic jackknife

for both groups), and removal of the blue and green subnetworks resulted in a comparable
(and near-zero) change in modularity across the groups (Figura 3, right panel).

Interpretation. The red subnetwork was highly integrated in both Group A and Group B;
Tuttavia, its removal led to less of an impact on modularity in Group A than Group B. IL
black and yellow subnetworks were more integrated into the rest of the network in Group A
than Group B.

Comparing group differences with differential impact analysis permits one to directly assess
the impact that local differences can have on global ﬁndings. To summarize, the removal of
the red subnetwork corresponded to a signiﬁcant group difference as well as a signiﬁcant
differential impact, suggesting that its removal led to the networks becoming more different
from each other. In contrasto, the removal of the yellow or black subnetworks led to signiﬁcant
differential impact but no signiﬁcant group differences, indicating that the removal of either of
these subnetworks led to the networks becoming more similar to one another.

Simulation Study: Speciﬁcity and Sensitivity of the NS Jackknife

A simulation study assessing the speciﬁcity and sensitivity of the NS jackknife to the magni-
tude of group differences, sample size, and network size was conducted. Results show that NS
jackknife has good sensitivity to detect small group differences in the generating model at mod-
erate sample sizes (50 subjects per group) and larger networks (250 nodes) and shows excel-
lent sensitivity for larger group differences across multiple conditions. In a condition in which
groups had different whole-network statistics that were driven by speciﬁc local differences,
NS jackknife detected differences in differential impact across all subnetworks (Condition 4,
Supplementary Figure 5), and an examination of effect sizes correctly identiﬁed which subnet-
works had the most impact on the network statistic (Supplementary Figure 6). In all conditions
tested, network size was an important predictor of power, with larger networks demonstrating
greater power to detect small effects.

One key conclusion from this simulation study regards the interpretation of NS jackknife
results when the tested groups are signiﬁcantly different in their whole-brain statistic. In this
case, the NS jackknife tends to detect differences upon the removal of all the network elements
being jackknifed on. In questo caso, two routes of interpretation are viable. The ﬁrst is to examine
the effect sizes of differences to determine which component of the network contributes most
to the differences between the groups (cioè., which has the largest effect size). The second is to
determine whether any subnetwork was not found to have a signiﬁcant differential impact, In
which case it can be concluded that those subnetworks were not signiﬁcant contributors to
the observed group difference.

For full details of this simulation study, see the Supporting Information.

Empirical Example: Reduced Integration of Subnetworks in ADHD

This empirical example is a proof of concept demonstrating how the use of the NS jackknife
provides important information about differences in functional brain network organization in
children with ADHD and TD children. ADHD is a neurodevelopmental disorder characterized
by difﬁculty with sustaining attention, as well as excessive impulsive or hyperactive behavior
(American Psychiatric Association, 2013). Existing research indicates that whole-brain func-
tional and structural topology are disrupted in individuals with ADHD, as are speciﬁc local
differences in connectivity (for reviews, see the following: Cao, Shu, Cao, Wang, & Lui, 2014;

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Henry & Cohen, 2019). Individuals with ADHD exhibit higher whole-brain functional segrega-
tion than healthy control participants (Lin et al., 2014; Wang et al., 2009). In terms of speciﬁc
local differences, there appears to be disruptions in networks such as the default mode, fron-
toparietal, Attenzione, visual, and motor subnetworks (Castellanos & Proal, 2012). Given these
ﬁndings at the global and local level, a comparison of individuals with ADHD and healthy
control participants is a useful test case for the NS jackknife. In the following analyses, we
examine global efﬁciency (Latora & Marchiori, 2001) of resting-state functional MRI scans in
children with ADHD as compared with TD children. Global efﬁciency is a measure of func-
tional integration. Previous studies have mixed ﬁndings with regard to differences in global
efﬁciency, either showing that individuals with ADHD have slightly lower global efﬁciency
than healthy control participants (Lin et al., 2014) or no differences in global efﬁciency (Wang
et al., 2009). In this example, we attempt to localize potential differences in global efﬁciency
to speciﬁc functional subnetworks in an effort to learn more about these inconsistencies in the
prior literature.

METHODS

Sample Characteristics and Data Acquisition

Participants included 80 children with ADHD between 8 E 12 years of age (male N = 55
and female N = 25) E 208 age-matched TD children (male N = 150 and female N = 58).
For details on study inclusion criteria, see Barber and colleagues (2015).

All data were collected at the Kennedy Krieger Institute (Baltimore, MD) using a Philips
3T Achieva MRI scanner. High-resolution T1-weighted 3D MPRAGE images were acquired
with the following parameters: repetition time (TR) = 8.05 ms, echo time (TE) = 3.76 ms, ﬂip
angle = 8◦, matrix 256 × 256, ﬁeld of view (FOV) = 200 mm, and slice thickness 1 mm. For
resting-state data, 21 participants had 128 T2*-weighed echoplanar images (EPIs), while 167
participants had 156, both collected with the following parameters: TR = 2.5 S, TE = 30 ms,
ﬂip angle = 70◦, matrix 96 × 96, and FOV = 256 mm.

Preprocessing

The resting-state images were preprocessed using FMRIPREP version 1.0.7 (Esteban et al.,
2018), a Nipype-based tool (Gorgolewski et al., 2011). Each T1-weighted volume was cor-
rected for intensity nonuniformity using N4BiasFieldCorrection v2.1.0 (Tustison et al., 2010)
and skull-stripped using antsBrainExtraction.sh v2.1.0 (using the OASIS template). Brain sur-
faces were reconstructed using recon-all from FreeSurfer v6.0.1 (Dale, Fischl, & Sereno, 1999),
and the brain mask estimated previously was reﬁned with a custom variation of the method
to reconcile ANTs-derived and FreeSurfer-derived segmentations of the cortical graymatter of
Mindboggle (Klein et al., 2017). Spatial normalization to the ICBM 152 Nonlinear Asymmet-
ric template version 2009c (Fonov, Evans, McKinstry, Almli, & Collins, 2009) was performed
through nonlinear registration with the antsRegistration tool of ANTs v2.1.0 (Avants, Epstein,
Grossman, & Gee, 2008), using brain-extracted versions of both the T1-weighted volume
and template. Brain tissue segmentation of cerebrospinal ﬂuid (CSF), whitematter (WM), E
graymatter (GM) was performed on the brain-extracted T1-weighted image using FAST
(Zhang, Brady, & Smith, 2001).

Functional data were slice time corrected using 3dTshift from AFNI v16.2.07 (Cox, 1996)
and motion corrected using mcﬂirt (FSL v5.0.9; Jenkinson, Bannister, Brady, & Smith, 2002).

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

This was followed by coregistration to the corresponding T1-weighted image using boundary-
based registration (Greve & Fischl, 2009) con 9 degrees of freedom, using bbregister (FreeSurfer
v6.0.1). Motion-correcting transformations, BOLD-to-T1-weighted transformation, and T1-
weighted-to-template (MNI) warp were concatenated and applied in a single step using
antsApplyTransforms (ANTs v2.1.0) using Lanczos interpolation. Frame-wise displacement
(FD) (Power et al., 2013) was calculated for each functional run using the implementation
of Nipype.

Many internal operations of FMRIPREP use Nilearn (Abraham et al., 2014), principally
within the BOLD-processing workﬂow. For more details of the pipeline see https://fmriprep.
readthedocs.io/en/latest/workﬂows.html. This description of the preprocessing pipeline was
generated at http://fmriprep.readthedocs.io/en/latest/citing.html.

Motion Processing and Spectral Filtering

Following preprocessing with FMRIPREP, the resting-state scans were corrected for nuisance
variables, including motion, WM signal, and CSF signal. This correction proceeded in the
following steps. Timepoints that exceeded 0.3mm FD (Energia, Barnes, Snyder, Schlaggar, &
Petersen, 2012) were ﬂagged, as were the timepoints immediately preceding and following the
ﬂagged timepoint. Interpolation across these ﬂagged timepoints was conducted using spectral
interpolation (Ciric et al., 2017). Following the spectral interpolation, a high-pass ﬁlter at 0.008
MHz was applied to the voxel time series containing the interpolated timepoints. The ﬂagged
timepoints were then set to missing for all voxels. A nuisance regression set consisting of the 6
degrees of motion and mean WM and CSF signal, as well as the temporal derivatives, quadratic
expansions, and the quadratic expansions of the temporal derivatives of these 8 regressors, era
high-pass ﬁltered at 0.008 MHz, as per the suggestion of Ciric, Satterthwaite, and colleagues
(Ciric et al., 2017; Satterthwaite et al., 2013). The ﬁltered nuisance regressors were used to
regress out nuisance signal from the scrubbed and ﬁltered voxel time series. Finalmente, subjects
that had more than 30% of their data lost because of scrubbing were removed from the analysis,
as were subjects who exceeded 0.2 mm mean FD overall. This resulted in a ﬁnal sample of
34 children with ADHD and 121 TD children, with mean percentage timepoints lost 8% E
10% rispettivamente. This difference in percentage timepoints lost was nonsigniﬁcant (T(48.423) =
−1.47, p = 0.14). Additionally, the mean FD in the ﬁnal sample was not signiﬁcantly different
between children with ADHD and TD children (T(49.902) = 1.27, p = 0.21, ADHD mean =
0.131, TD mean = 0.121).

ROI Extraction and Network Construction

From the preprocessed and motion-corrected resting-state images, ROI time series were
extracted using a commonly used functional brain atlas (Power et al., 2011). This atlas
consists of 264 ROIs, 234 of which are classiﬁed into 13 functional subnetworks. These
functional subnetworks and abbreviations are as follows: auditory (AUD), cerebellar (CER),
cingulo-opercular (CO), default mode (DMN), dorsal attention (DAN), frontoparietal (FP),
memory (MEM), salience (SAL), somatomotor hand (SM Hand), somatomotor mouth (SM
Mouth), subcortical (SUB), ventral attention (VAN), and visual (VIS). From these ROI time series,
234 × 234 correlation matrices were computed. These correlation matrices were then con-
verted to absolute values and thresholded at a correlation magnitude of 0.35 to create binary
connectivity networks. The pattern of ﬁndings below was consistent for a band of thresholds
from 0.25 A 0.4.

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Network Statistics and Jackknife

In this application, we use global efﬁciency as our network statistic. Global efﬁciency is de-
ﬁned as the average of the inverse shortest path length between all nodes in the network (Latora
& Marchiori, 2001), and can be thought of as a measure of global integration. We applied the
jackknife over the 13 functional subnetworks in the functional brain atlas. The two tests previ-
ously described were then applied to the jackknifed data. This allowed us to assess (UN) whether
the removal of a functional subnetwork led to a signiﬁcant difference in global efﬁciency be-
tween children with ADHD and TD children; E (B) whether the impact of the removal of
a functional subnetwork on functional integration was different for children with ADHD as
compared with TD children.

Jackknife Estimate and Group Comparisons

Independent two-sample t tests were conducted to compare global efﬁciency values across the
entire sample, and to compare global efﬁciency values and differential impacts across groups.
Differential impact results were corrected for multiple dependent comparisons using the BH
correction, as described in the above simulations.

RESULTS

Group differences in whole-brain global efﬁciency: There was no group difference in global
efﬁciency for children with ADHD and TD children (T(5037) = 0.99, p = 0.32; ADHD
mean = 0.29, TD mean = 0.28).

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Figura 4. Group differences in global efﬁciency between children with ADHD and TD children
after the removal of a given subnetwork. Networks are presented here in alphabetical order. Central
line is the median, box is the interquartile range (IQR), and whiskers represent 1.5 ∗ IQR. Points
represent values more than 2 SD away from the mean. Removal of the SUB network resulted in
children with ADHD having signiﬁcantly higher global efﬁciency than TD children.

Network Neuroscience

The network statistic jackknife

Group differences by subnetwork removal: Prossimo, we examined group differences in global
efﬁciency between children with ADHD and TD children after removing each functional
subnetwork.

Removal of the SUB subnetwork led to a signiﬁcant difference between children with ADHD
and TD children with respect to global efﬁciency (Figura 4). Speciﬁcally, children with ADHD
had greater global efﬁciency than TD children after SUB subnetwork removal. Tavolo 1 in the
Supporting Information contains the numerical differences and p values. It should be noted that
these results are best interpreted when combined with the differential impact results below.

Differential impact of subnetwork removal: Finalmente, we examined whether the removal of
any of the functional subnetworks led to a differential change in global efﬁciency for children
with ADHD as compared with TD children.

The only subnetwork that exhibited signiﬁcant differential impact after adjusting for mul-
tiple comparisons was the SUB subnetwork. Children with ADHD exhibited an increase in
global efﬁciency after SUB subnetwork removal, while TD children exhibited a slight decrease
(Figura 5). These ﬁndings suggest that the SUB subnetwork has lower integration with the global
network in children with ADHD. The lower integration of the SUB subnetwork in children with
ADHD is consistent with prior ﬁndings of disruptions in cortico-striatal-thalamic-cortical loops
in ADHD (for a review, see Posner, Park, & Wang, 2014). Importantly, this ﬁnding is an ex-
ample of equiﬁnality, as the difference in the local topology of the subcortical network did
not lead to an observed difference in whole-brain global efﬁciency. Tavolo 3 in the Supporting
Information contains the numerical differences and p values for this analysis.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Figura 5. Differential impact on global efﬁciency when removing subnetworks between children
with ADHD and TD children. Central line is the median, box is the interquartile range (IQR), E
whiskers represent 1.5 ∗ IQR. Points represent values more than 2 SD away from the mean. Removal
of the subcortical subnetwork (SUB) led to a greater increase in global efﬁciency in children with
ADHD as compared with TD children, suggesting that the SUB subnetwork is less integrated into the
whole-brain network in children with ADHD. Signiﬁcant differences were based on BH corrected
p values.

Network Neuroscience

The network statistic jackknife

DISCUSSION

Whole-brain network analysis is a powerful tool for assessing brain topology, yet it lacks the
ability to localize ﬁndings related to global topology to a speciﬁc location. This issue, com-
bined with the fact that robust local differences do not necessarily result in observable global
differences in topology, suggests that whole-brain network analysis has a substantial blind
spot. Local graph theory analysis metrics can alleviate this but lack the ability to directly relate
local ﬁndings to global topology. In questo articolo, we introduced the NS jackknife, a ﬂexible tool
for localizing global differences in network topology. Our framework links the global scope to
the local scope (Hallquist & Hillary, 2018) and is ﬂexible enough to operate on any global
network statistic. We implemented this framework in an open source R package, netjack
to make it accessible to re-
(Henry, 2018; https://CRAN.R-project.org/package=netjack)
searchers. Our simulation study evaluating the performance of the NS jackknife showed that
the method has excellent sensitivity to local differences in network topology across a variety of
conditions, while keeping the false discovery rate at a nominal level (0.05; see the Supporting
Information).

The NS jackknife is an extension of the simulated lesioning approach that offers a variety of
advantages to neuroimaging researchers interested in examining group differences in network
topology. The NS jackknife extends the simulated lesioning approach beyond its most common
application, examining the robustness of a network to attack, thus enabling researchers to
examine how individual elements of a given network contribute to a global network statistic.
It can be applied to any global network statistic under study, which allows researchers to extend
existing analysis pipelines rather than requiring implementation of a new set of analyses. IL
speciﬁcity of a global analysis can be tested at any level of resolution (subnetwork, node,
edge), and resulting signiﬁcance tests are appropriately corrected for multiple comparisons,
making this an easy tool to implement and interpret. Finalmente, while the focus here was on binary
networks derived from functional MRI data, the NS jackknife can easily be applied to weighted
functional networks or to structural networks derived from diffusion-weighted imaging.

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

There are several further methodological developments for the NS jackknife. While we fo-
cused on comparisons between two groups of subjects, future versions of the NS jackknife
toolbox will allow for within-subject across-time comparisons, testing covariates and control-
ling for potential confounds, and analyzing factorial grouping structures. The NS jackknife
framework can be extended to generalized linear regression approaches as well as to allow
for dependencies, such as those that arise from multiple collection sites. This is an active area
of development, with implementations being introduced in the near future. Although the NS
jackknife as currently presented examines network statistics as the outcome, it is likewise nat-
ural to consider network statistics as predictors of some distal outcome, such as behavior or
symptomology. Our framework can be extended to handle cases where some distal outcome
is of interest and would allow an assessment of the importance of local differences in network
topology with regard to that distal outcome.

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

Finalmente, our empirical example produced results that were consistent with prior results
suggesting little difference in global metrics of functional integration (cioè., global efﬁciency)
between children with ADHD and TD children. Importantly, our use of the NS jackknife un-
covered an occurrence of equiﬁnality, in which a local group difference did not lead to a
global group difference. Speciﬁcally, an analysis of group differences and differential impact of
subnetwork removal showed that the subcortical subnetwork was less integrated into the whole-
brain network in children with ADHD, consistent with prior literature (Hong et al., 2014).

Network Neuroscience

The network statistic jackknife

Limitations and Considerations

A key consideration for the use of the NS jackknife is in the use of weighted networks or
networks formed from white matter tractography. The NS jackknife is agnostic to the type of
rete (weighted vs. binary, undirected vs. directed), but special care needs to be taken
when using weighted networks to ensure that the appropriate network statistics are used. For
esempio, when global efﬁciency is computed on correlation networks, it often exhibits scaling
issues due to taking the inverse of correlations that are close to zero. If those correlations are
removed during the NS jackknife, the resulting difference in global efﬁciency could simply be
due to that scaling, and not due to any true difference in network structure. This is an issue for
any weighted graph analysis and is not limited to the NS jackknife approach.

NS jackknife as a general framework has a limitation that users need to consider. As most
tests of interest involve multiple comparisons, a multiple comparison correction is necessary to
reduce the false discovery rate. Because of this, as the number of network features jackknifed
increases, power to detect any given difference due to the removal of a single network feature
decreases. This therefore limits the scope of an NS jackknife application. In smaller samples,
NS jackknife is likely best applied to questions regarding functional subnetworks, in which
the number of subnetworks is relatively small and potential impacts are large. For larger
studies, particularly studies utilizing public datasets such as the Human Connectome Project
(Van Essen et al., 2013) or the Autism Brain Imaging Data Exchange (ABIDE) datasets
(Di Martino et al., 2017; Di Martino et al., 2014), it becomes more feasible to examine dif-
ferences at the level of nodes or even edges. Another strategy that researchers with smaller
samples can employ is examining more localized networks with the NS jackknife. For ex-
ample, conducting an analysis limiting the network under consideration to the default mode
network and removing nodes and edges of that smaller network rather than of the whole-brain
rete. This particular limitation, Tuttavia, is inherent to neuroimaging analysis given the
high-dimensional nature of the data. This speaks to a general need for the development of
analytic methods of high-dimensional data dealing with issues of global to local scoping as
discussed above. One potential solution to this problem is to use the screen ﬁltering method
of Meskaldji and colleagues (2015) to adjust the p values for tests of difference at the node or
edge level with respect to subnetwork groupings.

For pedagogical purposes the manuscript focused on detecting local differences in single
canonical functional subnetworks; Tuttavia, the NS jackknife can be applied to combinations
of canonical subnetworks as well combinations of nodes or edges. One key consideration
here is that of combinatoric complexity. When using the NS jackknife to test removal of com-
binations of multiple subnetworks or nodes, the number of possible combinations increases
factorially. To avoid this, the NS jackknife can be used in a conﬁrmatory way, with researchers
specifying a set of combinations of network structures informed by previous literature or a pre-
vious exploratory application of NS jackknife, then testing for local differences with that set
of combinations. Finalmente, the NS jackknife does not require the use of canonical subnetworks
as demonstrated here. Sample-speciﬁc parcellations derived from some other method can be
used instead. The only requirement here is that a given network is deﬁned on the same nodes
for all subjects.

There are two additional considerations for researchers interested in using the NS jackknife
framework. The ﬁrst is that of network construction. The NS jackknife assumes a ﬁxed network
structure, and that the only changes that are occurring are due to the removal of speciﬁc
network features. This means that network estimation techniques, such as regularized partial
correlations, are to be used before the NS jackknife is applied, and not applied to the jackknifed

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

networks themselves. This ensures that any localized effect can be interpreted as due to the
removal of the speciﬁc structure only, and not an analytical artifact caused by a reestimation
of the network.

The second consideration is that of the density/strength dependency. As mentioned earlier,
network statistics (such as average degree) are dependent on the size and density of the network
(van Wijk et al., 2010). As such, it is expected that the network statistics would change when
structures are removed. This potentially causes issues with interpretation, where an observed
group difference might simply be due to differences in density of the removed component
rather than any topological difference. One potential solution to this interpretational concern
is to add the network element’s density or strength as a covariate to the jackknife models.
Research into adjustments for density/strength dependency is ongoing.

CONCLUSION

In closing, the NS jackknife framework bridges the gap between global network analysis and
local differences in topology, making it an ideal tool with which to investigate the speciﬁcity
of observed differences in whole-brain topology. Inoltre, it allows researchers to explore
local network features in the absence of a global result while still using the tools, terminology,
and interpretation of network neuroscience at the global level. Finalmente, future developments will
extend the NS jackknife to more complex modeling frameworks and settings, further extending
its utility within the neuroimaging community.

SUPPORTING INFORMATION

Supporting information for this article is available at https://doi.org/10.1162/netn_a_00109.
The NS jackknife method is available at https://cran.r-project.org/package=netjack (Henry,
2018).

AUTHOR CONTRIBUTIONS

Teague Henry: Formal analysis; Investigation; Methodology; Software; Visualization; Writing –
Original Draft; Writing – Review & Editing. Kelly A. Duffy: Data curation; Resources; Writing –
Review & Editing. Marc D. Rudolph: Writing – Original Draft; Writing – Review & Editing. Mary
Beth Nebel: Data curation; Writing – Review & Editing. Stewart H. Mostofsky: Data curation;
Funding acquisition; Writing – Review & Editing. Jessica R. Cohen: Conceptualization; Funding
acquisition; Resources; Supervision; Writing – Original Draft; Writing – Review & Editing.

FUNDING INFORMATION

Jessica R. Cohen, National Institute of Mental Health (US), Award ID: R00MH102349. Stewart
H. Mostofsky, National Institute of Mental Health (US), Award ID: R01MH085328. Stewart
H. Mostofsky, National Institute of Mental Health (US), Award ID: R01MH078160. Mary Beth
Nebel, National Institute of Mental Health (US), Award ID: K01MH109766.

REFERENCES

Abraham, A., Pedregosa, F., Eickenberg, M., Gervais, P., Mueller,
A., Kossaiﬁ, J., . . . Varoquaux, G. (2014). Machine learning for
neuroimaging with scikit-learn. Frontiers in Neuroinformatics, 8.
https://doi.org/10.3389/fninf.2014.00014

Achard, S., Salvador, R., Whitcher, B., Suckling, J., & Bullmore,
E. (2006). A resilient, low-frequency, small-world human brain
functional network with highly connected association cortical

hub. Journal of Neuroscience, 26(1), 63–72. https://doi.org/10.
1523/JNEUROSCI.3874-05.2006

Alstott, J., Breakspear, M., Hagmann, P., Cammoun, L., & Sporns,
O. (2009). Modeling the impact of lesions in the human brain.
PLoS Computational Biology, 5(6), e1000408. https://doi.org/10.
1371/journal.pcbi.1000408

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

American Psychiatric Association. (2013). Diagnostic and statistical
manual of mental disorders (5th ed.). Arlington, VA: American
Psychiatric Publishing.

Avants, B. B., Epstein, C. L., Grossman, M., & Gee, J. C. (2008). Sym-
metric diffeomorphic image registration with cross-correlation:
Evaluating automated labeling of elderly and neurodegenerative
brain. Medical Image Analysis, 12(1), 26–41. https://doi.org/10.
1016/j.media.2007.06.004

Barber, UN. D., Jacobson, l. A., Wexler, J. L., Nebel, M. B., Caffo,
B. S., Pekar, J. J., & Mostofsky, S. H. (2015). Connectivity support-
ing attention in children with attention deﬁcit hyperactivity dis-
order. NeuroImage: Clinical, 7, 68–81. https://doi.org/10.1016/j.
nicl.2014.11.011

Bassett, D. S., & Sporns, O. (2017). Network neuroscience. Nature
Neuroscience, 20(3), 353–364. https://doi.org/10.1038/nn.4502
Baum, G. L., Ciric, R., Roalf, D. R., Betzel, R. F., Moore, T. M.,
(2017). Modular
Shinohara, R. T.,
segregation of structural brain networks supports the develop-
ment of executive function in youth. Current Biology, 27(11),
1561–1572.e8. https://doi.org/10.1016/j.cub.2017.04.051

. Satterthwaite, T. D.

Benjamini, Y., & Hochberg, Y. (1995). Controlling the false discov-
ery rate: A practical and powerful approach to multiple testing.
Journal of the Royal Statistical Society. Series B (Methodological),
57(1), 289–300. https://doi.org/10.2307/2346101

Benjamini, Y., & Yekutieli, D. (2001). The control of the false discov-
ery rate in multiple testing under dependency. Annals of Statistics,
29(4), 1165–1188. https://doi.org/10.1214/aos/1013699998
Bertalanffy, l. Von. (1950). An outline of general system theory.
British Journal for the Philosophy of Science, 1(2), 134–165. https://
doi.org/10.1093/bjps/I.2.134

Cao, M., Shu, N., Cao, Q., Wang, Y., & Lui, Y. (2014). Imag-
ing functional and structural brain connectomics in attention-
deﬁcit/hyperactivity disorder. Molecular Neurobiology, 50(3),
1111–1123. https://doi.org/10.1007/s12035-014-8685-x

Castellanos, F. X., & Proal, E. (2012). Large-scale brain systems in
ADHD: Beyond the prefrontal-striatal model. Trends in Cogni-
tive Sciences, 16(1), 17–26. https://doi.org/10.1016/j.tics.2011.
11.007

Ciric, R., Wolf, D. H., Energia, J. D., Roalf, D. R., Baum, G. L.,
Ruparel, K., . . . Satterthwaite, T. D. (2017). Benchmarking of
participant-level confound regression strategies for the control
of motion artifact in studies of functional connectivity. Neuro-
Image,154, 174–187. https://doi.org/10.1016/j.neuroimage.2017.
03.020

Cohen, J. R., & D’Esposito, M. (2016). The segregation and integra-
tion of distinct brain networks and their relationship to cognition.
Journal of Neuroscience, 36(48), 12083–12094. https://doi.org/
10.1523/JNEUROSCI.2965-15.2016

Cox, R. W. (1996). AFNI: Software for analysis and visualization
of functional magnetic resonance neuroimages. Computers and
Biomedical Research, 29(3), 162–173. https://doi.org/10.1006/
cbmr.1996.0014

Crucitti, P., Latora, V., Marchiori, M., & Rapisarda, UN. (2004). Error
and attack tolerance of complex networks. Physica A: Statistical
Mechanics and Its Applications, 340(1–3), 388–394. https://doi.
org/10.1016/j.physa.2004.04.031

Dale, UN. M., Fischl, B., & Sereno, M. IO. (1999). Cortical surface-
based analysis:
IO. Segmentation and surface reconstruction.
NeuroImage, 9(2), 179–194. https://doi.org/10.1006/nimg.1998.
0395

Di Martino, A., O’Connor, D., Chen, B., Alaerts, K., Anderson, J. S.,
Assaf, M., . . . Milham, M. P. (2017). Enhancing studies of the
connectome in autism using the autism brain imaging data ex-
change II. Scientiﬁc Data, 4(170010). https://doi.org/10.1038/
sdata.2017.10

Di Martino, A., Shehzad, Z., Kelly, C., Roy, UN. K., Gee, D. G., Uddin,
l. Q., . . . Milham, M. P. (2009). Relationship between cingulo-
insular functional connectivity and autistic traits in neurotypical
adults. American Journal of Psychiatry, 166(8), 891–899. https://
doi.org/10.1176/appi.ajp.2009.08121894

Di Martino, A., Yan, C. G.-G., Li, Q., Denio, E., Castellanos,
F. X., Alaerts, K., . . . Milham, M. P. (2014). The autism brain
imaging data exchange: Towards a large-scale evaluation of the
intrinsic brain architecture in autism. Molecular Psychiatry, 19(6),
659–667. https://doi.org/10.1038/mp.2013.78

Efron, B. (1982). The jackknife, the bootstrap and other resampling
plans. Philadelphia, PAPÀ: Society for Industrial and Applied Math-
ematics. https://doi.org/10.1137/1.9781611970319

Esteban, O., Markiewicz, C., Blair, R. W., Moodie, C., Isik, UN. I.,
Aliaga, UN. E., . . . Gorgolewski, K. J. (2018). FMRIPrep: A ro-
bust preprocessing pipeline for functional MRI. BioRxiv, 306951.
https://doi.org/10.1101/306951

Fonov, V., Evans, A., McKinstry, R., Almli, C., & Collins, D. (2009).
Unbiased nonlinear average age-appropriate brain templates
from birth to adulthood. NeuroImage, 47(1), S102. https://doi.
org/10.1016/S1053-8119(09)70884-5

Fornito, A., Bullmore, E. T., & Zalesky, UN. (2017). Opportunities and
challenges for psychiatry in the connectomic era. Biological Psy-
chiatry: Cognitive Neuroscience and Neuroimaging, 2(1), 9–19.
https://doi.org/10.1016/j.bpsc.2016.08.003

Fornito, A., Zalesky, A., & Breakspear, M. (2015). The connectomics of
brain disorders. Nature Reviews Neuroscience, 16(3), 159–172.
https://doi.org/10.1038/nrn3901

Gorgolewski, K., Burns, C. D., Madison, C., Clark, D., Halchenko,
Y. O., Waskom, M. L., & Ghosh, S. S. (2011). Nipype: A ﬂexible,
lightweight and extensible neuroimaging data processing frame-
work in Python. Frontiers in Neuroinformatics, 5(13). https://doi.
org/10.3389/fninf.2011.00013

Grayson, D. S., & Fair, D. UN. (2017). Development of large-scale
functional networks from birth to adulthood: A guide to the neu-
roimaging literature. NeuroImage, 160, 15–31. https://doi.org/
10.1016/j.neuroimage.2017.01.079

Greicius, M. D., Krasnow, B., Reiss, UN. L., & Menon, V. (2003).
Functional connectivity in the resting brain: A network analy-
sis of the default mode hypothesis. Proceedings of the National
Academy of Sciences, 100(1), 253–258. https://doi.org/10.1073/
pnas.0135058100

Greve, D. N., & Fischl, B. (2009). Accurate and robust brain image
alignment using boundary-based registration. NeuroImage, 48(1),
63–72. https://doi.org/10.1016/j.neuroimage.2009.06.060

Guimerà, R., & Nunes Amaral, l. UN. (2005). Functional cartography
of complex metabolic networks. Nature, 433(7028), 895–900.
https://doi.org/10.1038/nature03288

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Hallquist, M. N., & Hillary, F. G. (2018). Graph theory approaches
to functional network organization in brain disorders: A critique
for a brave new small-world. Network Neuroscience, 1(agosto),
243741. https://doi.org/10.1101/243741

Henry, T. R.

(2018). Netjack: Tools for working with samples
of networks. Retrieved from https://cran.r-project.org/package=
netjack

Henry, T. R., & Cohen, J. R. (2019). Dysfunctional brain network
organization in neurodevelopmental disorders. In P. J. Laurienti,
B. Munsell, & G. Wu (Eds.), Connectomics: Methods, mathemat-
ical models and applications. Elsevier.

Henry, T. R., Dichter, G. S., & Gates, K. (2018). Age and gender
effects on intrinsic connectivity in autism using functional in-
tegration and segregation. Biological Psychiatry: Cognitive Neu-
roscience and Neuroimaging, 3(5), 414–422. https://doi.org/10.
1016/j.bpsc.2017.10.006

Hong, S. B., Zalesky, A., Fornito, A., Park, S., Yang, Y. H., Park,
M. H., . . . Kim, J. W. (2014). Connectomic disturbances in
attention-deﬁcit/hyperactivity disorder: A whole-brain tractogra-
phy analysis. Biological Psychiatry, 76(8), 656–663. https://doi.
org/10.1016/j.biopsych.2013.12.013

Jenkinson, M., Bannister, P., Brady, M., & Smith, S. (2002). Improved
optimization for the robust and accurate linear registration and
motion correction of brain images. NeuroImage, 17(2), 825–841.
https://doi.org/10.1016/S1053-8119(02)91132-8

Klein, A., Ghosh, S. S., Bao, F. S., Giard, J., Häme, Y., Stavsky,
E., . . . Keshavan, UN. (2017). Mindboggling morphometry of hu-
man brains. PLoS Computational Biology, 13(2). https://doi.org/
10.1371/journal.pcbi.1005350

Kucyi, A., Tambini, A., Sadaghiani, S., Keilholz, S., & Cohen, J. R.
(2018). Spontaneous cognitive processes and the behavioral vali-
dation of time-varying brain connectivity. Network Neuroscience,
1–57. https://doi.org/10.1162/NETN_a_00037

Latora, V., & Marchiori,M. (2001). Efficient behavior of small-worldnet-
works. Physical Review Letters, 87(19), 198701-1-198701–198704.
https://doi.org/10.1103/PhysRevLett.87.198701

Lewis, J. D., Theilmann, R. J., Townsend, J., & Evans, UN. C. (2013).
Network efﬁciency in autism spectrum disorder and its relation to
brain overgrowth. Frontiers in Human Neuroscience, 7. https://
doi.org/10.3389/fnhum.2013.00845

Lin, P., Sun, J., Yu, G., Wu, Y., Yang, Y., Liang, M., & Liu, X. (2014).
Global and local brain network reorganization in attention-
deﬁcit/hyperactivity disorder. Brain Imaging and Behavior, 8(4),
558–569. https://doi.org/10.1007/s11682-013-9279-3

Meskaldji, D. E., Vasung, L., Romascano, D., Thiran, J. P., Hagmann,
P., Morgenthaler, S., & Van De Ville, D. (2015). Improved stat-
istical evaluation of group differences in connectomes by
screening-ﬁltering strategy with application to study matura-
tion of brain connections between childhood and adolescence.
NeuroImage,108, 251–264. https://doi.org/10.1016/j.neuroimage.
2014.11.059

Newman, M. E. J. (2006). Modularity and community structure in
networks. Proceedings of the National Academy of Sciences, 103(23),
8577–8582. https://doi.org/10.1073/pnas.0601602103

Nowicki, K., & Snijders, T. UN. B. (2001). Estimation and predic-
tion for stochastic blockstructures. Journal of the American Statis-

tical Association, 96(455), 1077–1087. https://doi.org/10.1198/
016214501753208735

Pievani, M., de Haan, W., Wu, T., Seeley, W. W., & Frisoni, G. B.
(2011). Functional network disruption in the degenerative de-
mentias. The Lancet Neurology, 10(9), 829–843,. https://doi.org/
10.1016/S1474-4422(11)70158-2

Posner, J., Park, C., & Wang, Z. (2014). Connecting the dots: UN
review of resting connectivity MRI studies in attention-deﬁcit/
hyperactivity disorder. Neuropsychology Review, 24(1), 3–15.
https://doi.org/10.1007/s11065-014-9251-z

Energia, J. D., Barnes, K. A., Snyder, UN. Z., Schlaggar, B. L., &
Petersen, S. E. (2012). Spurious but systematic correlations in
functional connectivity MRI networks arise from subject mo-
zione. NeuroImage, 59(3), 2142–2154. https://doi.org/10.1016/j.
neuroimage.2011.10.018

Energia, J. D., Cohen, UN. L., Nelson, S. M., Wig, G. S., Barnes, K. A.,
Church, J. A., . . . Petersen, S. E. (2011). Functional network or-
ganization of the human brain. Neuron, 72(4), 665–678. https://
doi.org/10.1016/j.neuron.2011.09.006

Energia, J. D., Mitra, A., Laumann, T. O., Snyder, UN. Z., Schlaggar,
B. L., & Petersen, S. E. (2013). Methods to detect, characterize,
and remove motion artifact in resting state fMRI. NeuroImage, 84,
320–341. https://doi.org/10.1016/j.neuroimage.2013.08.048
Roine, U., Roine, T., Salmi, J., Nieminen-von Wendt, T., Tani,
P., Leppämäki, S., . . . Sams, M. (2015). Abnormal wiring of
the connectome in adults with high-functioning autism spectrum
disorder. Molecular Autism, 6(1), 65. https://doi.org/10.1186/
s13229-015-0058-4

Rudie,

J. D., & Dapretto, M.

(2013). Convergent evidence of
brain overconnectivity in children with autism? Cell Reports, 5(3),
565–566. https://doi.org/10.1016/j.celrep.2013.10.043

J., Calkins, M. E.,

Satterthwaite, T. D., Elliott, M. A., Gerraty, R. T., Ruparel, K.,
Loughead,
(2013). An
.
improved framework for confound regression and ﬁltering for
control of motion artifact in the preprocessing of resting-state
functional connectivity data. NeuroImage, 64(1), 240–256. https://
doi.org/10.1016/j.neuroimage.2012.08.052

. Worf, D. H.

Seeley, W. W., Menon, V., Schatzberg, UN. F., Keller, J., Glover, G. H.,
Kenna, H., . . . Greicius, M. D. (2007). Dissociable intrinsic con-
nectivity networks for salience processing and executive control.
Journal of Neuroscience, 27(9), 2349–2356. https://doi.org/10.
1523/JNEUROSCI.5587-06.2007

Shine, J. M., & Poldrack, R. UN. (2017). Principles of dynamic
network reconﬁguration across diverse brain states. NeuroIm-
age, 180(Pt. B), 396–405. https://doi.org/10.1016/j.neuroimage.
2017.08.010

Snijders, T. UN. B., & Borgatti, S. P. (1999). Non-parametric standard
errors and tests for network statistics. Connections, 22(2), 1–10.
Sporns, O. (2013). Network attributes for segregation and integra-
tion in the human brain. Current Opinion in Neurobiology, 23(2),
162–171. https://doi.org/10.1016/j.conb.2012.11.015

Sporns, O., Chialvo, D. R., Kaiser, M., & Hilgetag, C. C. (2004).
Organization, development and function of complex brain net-
works. Trends in Cognitive Sciences, 8(9), 418–425. https://doi.
org/10.1016/j.tics.2004.07.008

Tukey, J. W. (1958). Bias and conﬁdence in not-quite large sample.

Annals of Mathematical Statistics, 29, 614.

Network Neuroscience

D
o
w
N
o
UN
D
e
D

F
R
o
M
H

T
T

:
/
/

D
io
R
e
C
T
.

io
T
.

e
D
tu
N
e
N
UN
R
T
io
C
e
–
P
D

F
/

4
1
7
0
1
9
2
0
5
3
8
N
e
N
_
UN
_
0
0
1
0
9
P
D

B
sì
G
tu
e
S
T

o
N
0
7
S
e
P
e
M
B
e
R
2
0
2
3

The network statistic jackknife

Tustison, N.

J., Avants, B. B., Cook, P. A., Zheng, Y., Egan,
A., Yushkevich, P. A., & Gee, J. C. (2010). N4ITK: Improved
N3 bias correction. IEEE Transactions on Medical Imaging, 29(6),
1310–1320. https://doi.org/10.1109/TMI.2010.2046908

Van Essen, D. C., Smith, S. M., Barch, D. M., Behrens, T. E. J.,
Yacoub, E., & Ugurbil, K. (2013). The WU-minn human connec-
tome project: An overview. NeuroImage, 80, 62–79. https://doi.
org/10.1016/j.neuroimage.2013.05.041

van Wijk, B. C. M., Stam, C. J., & Daffertshofer, UN. (2010). Com-
paring brain networks of different size and connectivity density
using graph theory. PLoS ONE, 5(10), e13701. https://doi.org/10.
1371/journal.pone.0013701

Wang, L., Zhu, C., Lui, Y., Zang, Y., Cao, Q., Zhang, H., . . . Wang,
Y. (2009). Altered small-world brain functional networks in chil-

dren with attention-deﬁcit/hyperactivity disorder. Human Brain
Mapping, 30(2), 638–649. https://doi.org/10.1002/hbm.20530
Yoo, K., Lee, P., Chung, M. K., Sohn, W. S., Chung, S. J., Na, D. L.,
. . . Jeong, Y. (2017). Degree-based statistic and center persistency
for brain connectivity analysis. Human Brain Mapping, 38(1),
165–181. https://doi.org/10.1002/hbm.23352

Zalesky, A., Fornito, A., & Bullmore, E. T. (2010). Network-based
statistic: Identifying differences in brain networks. NeuroImage,
53(4), 1197–1207. https://doi.org/10.1016/j.neuroimage.2010.
06.041

Zhang, Y. J., Brady, M., & Smith, S. E. (2001). Segmentation of brain
MR images through a hidden Markov random ﬁeld model and the
expectation-maximization algorithm. IEEE Transactions on Med-
ical Imaging, 20, 45–57.