Métodos
Large-scale directed network inference
with multivariate transfer entropy and
hierarchical statistical testing
Leonardo Novelli
1
, Patricia Wollstadt
4
, Pedro Mediano
1
Michael Wibral
, and Joseph T. Lizier
2,∗
3
,
1Centre for Complex Systems, Faculty of Engineering, The University of Sydney, Sídney, Australia
2Honda Research Institute Europe, Offenbach am Main, Alemania
3Computational Neurodynamics Group, Department of Computing, Imperial College London, Londres, Reino Unido
Campus Institute for Dynamics of Biological Networks, Georg-August University, Göttingen, Alemania
First authors contributed equally to this work.
∗
4
un acceso abierto
diario
Palabras clave: Neuroimaging, Directed connectivity, Effective network, Multivariate transfer entropy,
Information theory, Nonlinear dynamics, Statistical inference, Nonparametric tests
ABSTRACTO
Network inference algorithms are valuable tools for the study of large-scale neuroimaging
conjuntos de datos. Multivariate transfer entropy is well suited for this task, being a model-free measure
that captures nonlinear and lagged dependencies between time series to infer a minimal
directed network model. Greedy algorithms have been proposed to efficiently deal with
high-dimensional datasets while avoiding redundant inferences and capturing synergistic
efectos. Sin embargo, multiple statistical comparisons may inflate the false positive rate and are
computationally demanding, which limited the size of previous validation studies. El
algorithm we present—as implemented in the IDTxl open-source software—addresses these
challenges by employing hierarchical statistical tests to control the family-wise error rate and
to allow for efficient parallelization. The method was validated on synthetic datasets
involving random networks of increasing size (hasta 100 nodos), for both linear and
nonlinear dynamics. The performance increased with the length of the time series, alcanzando
consistently high precision, recordar, and specificity (>98% on average) para 10,000 tiempo
muestras. Varying the statistical significance threshold showed a more favorable
precision-recall trade-off for longer time series. Both the network size and the sample size are
one order of magnitude larger than previously demonstrated, showing feasibility for typical
EEG and magnetoencephalography experiments.
INTRODUCCIÓN
The increasing availability of large-scale, fine-grained datasets provides an unprecedented op-
portunity for quantitative studies of complex systems. Sin embargo, a shift toward data-driven
modeling of these systems requires efficient algorithms for analyzing multivariate time series,
which are obtained from observation of the activity of a large number of elements.
In the field of neuroscience, the multivariate time series typically obtained from brain record-
ings serve to infer minimal (effective) network models which can explain the dynamics of the
nodes in a neural system. The motivation for such models can be, por ejemplo, to describe a
causal network (Ay & Polani, 2008; Friston, 1994) or to model the directed information flow
Citación: Novelli, l., Wollstadt, PAG.,
Mediano, PAG., Wibral, METRO., & Lizier, j. t.
(2019). Large-scale directed network
inference with multivariate transfer
entropy and hierarchical statistical
pruebas. Neurociencia en red, 3(3),
827–847. https://doi.org/10.1162/
netn_a_00092
DOI:
https://doi.org/10.1162/netn_a_00092
Supporting Information:
https://doi.org/10.1162/netn_a_00092
https://github.com/pwollstadt/IDTxl
Recibió: 24 Enero 2019
Aceptado: 24 Abril 2019
Conflicto de intereses: Los autores tienen
declaró que no hay intereses en competencia
existir.
Autor correspondiente:
Leonardo Novelli
leonardo.novelli@sydney.edu.au
Editor de manejo:
Olaf Sporns
Derechos de autor: © 2019
Instituto de Tecnología de Massachusetts
Publicado bajo Creative Commons
Atribución 4.0 Internacional
(CC POR 4.0) licencia
La prensa del MIT
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
/
t
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
t
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
in the system (Vicente et al., 2011) in order to produce a minimal computationally equivalent
network (Lizier & Rubinov, 2012).
Information theory (Cover & tomás, 2005; shannon, 1948) is well suited for the latter
motivation of inferring networks that describe information flow as it provides model-free mea-
sures that can be applied at different scales and to different types of recordings. These mea-
sures, including conditional mutual information (Cover & tomás, 2005) and transfer entropy
(Schreiber, 2000), are based purely on probability distributions and are able to identify non-
linear relationships (Paluš et al., 1993). Más importante, information-theoretic measures al-
low the interpretation of the results from a distributed computation or information processing
perspectiva, by modeling the information storage, transfer, and modification within the system
(Lizier, 2013). Por lo tanto, information theory simultaneously provides the tools for building the
network model and the mathematical framework for its interpretation.
The general approach to network model construction can be outlined as follows: para cualquier
target process (element) in the system, the inference algorithm selects the minimal set of pro-
cesses that collectively contribute to the computation of the target’s next state. Every process
can be separately studied as a target, and the results can be combined into a directed network
describing the information flows in the system. This task presents several challenges:
The state space of the possible network models grows faster than exponentially with
respect to the size of the network;
Information-theoretic estimators suffer from the “curse of dimensionality” for large sets
of variables (paninski, 2003; Roulston, 1999);
In a network setting, statistical significance testing requires multiple comparisons. Este
results in a high false positive rate (type I errors) without adequate family-wise error rate
controls (Dickhaus, 2014) or a high false negative rate (type II errors) with naive control
procedures;
Nonparametric statistical testing based on shuffled surrogate time series is computation-
ally demanding but currently necessary when using general information-theoretic esti-
mators (Bossomaier et al., 2016; Lindner et al., 2011).
Several previous studies (Faes et al., 2011; Lizier & Rubinov, 2012; Sun et al., 2015; Vlachos
& Kugiumtzis, 2010) proposed greedy algorithms to tackle the first two challenges outlined
arriba (see a summary by Bossomaier et al., 2016, segundo 7.2). These algorithms mitigate the curse
of dimensionality by greedily selecting the random variables that iteratively reduce the uncer-
tainty about the present state of the target. The reduction of uncertainty is rigorously quantified
by the information-theoretic measure of conditional mutual information (CMI), which can also
be interpreted as a measure of conditional independence (Cover & tomás, 2005). In partic-
ular, these previous studies employed multivariate forms of the transfer entropy, eso es, estafa-
ditional and collective forms (Lizier et al., 2008, 2010). En general, such greedy optimization
algorithms provide a locally optimal solution to the NP-hard problem of selecting the most
informative set of random variables. An alternative optimization strategy—also based on con-
ditional independence—employs a preliminary step to prune the set of sources (Runge et al.,
2012, 2018). Despite this progress, the computational challenges posed by the estimation of
multivariate transfer entropy have severely limited the size of problems investigated in previous
validation studies in the general case of nonlinear estimators, Por ejemplo, Montalto et al.
(2014) usado 5 nodes and 512 muestras; Kim et al. (2016) usado 6 nodes and 100 muestras;
Runge et al. (2018) usado 10 nodes and 500 muestras. Sin embargo, modern neural recordings
often provide hundreds of nodes and tens of thousands of samples.
Neurociencia en red
828
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
t
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
IDTxl:
The “Information Dynamics Toolkit
xl” is an open-source Python
package available on GitHub (ver
Supporting Information).
Markovian with finite memory:
The present state of the target does
not depend on the past values of the
target and the sources beyond a
maximum finite lag l
METRO.
These computational challenges, as well as the multiple testing challenges described above,
are addressed here by the implementation of rigorous statistical tests, which represent the main
theoretical contribution of this paper. These tests are used to control the family-wise error rate
and are compatible with parallel processing, allowing the simultaneous analysis of the targets.
This is a crucial feature, which enabled an improvement on the previous greedy algorithms.
Exploiting the parallel computing capabilities of high-performance computing clusters and
graphics processing units (GPUs) enabled the analysis of networks at a relevant scale for brain
recordings—up to 100 nodes and 10,000 muestras. Our algorithm has been implemented in
the recently released IDTxl Python package (the “Information Dynamics Toolkit xl”; Wollstadt
et al., 2019).
We validated our method on synthetic datasets involving random structural networks of
increasing size (also referred to as ground truth) and different types of dynamics (vector auto-
regressive processes and coupled logistic maps). En general, effective networks are able to re-
flect dynamic changes in the regime of the system and do not reflect an underlying structural
network. Sin embargo, in the absence of hidden nodes (and other assumptions, including sta-
tionarity and the causal Markov condition), the inferred information network was proven to
reflect the underlying structure for a sufficiently large sample size (Sun et al., 2015). Experi-
ments under these conditions provide arguably the most important validation that the algorithm
performs as expected, and here we perform the first large-scale empirical validation for non-
Gaussian variables. As shown in the Results, the performance of our algorithm increased with
the length of the time series, reaching consistently high precision, recordar, and specificity (>98%
on average) para 10,000 time samples. Varying the statistical significance threshold showed a
more favorable precision-recall trade-off for longer time series.
MÉTODOS
Definitions and assumptions
Let us consider a system of N discrete-time stochastic processes for which a finite number of
samples have been recorded (over time and/or in different replications of the same experiment).
En general, let us assume that the stochastic processes are stationary in each experimental
time-window and Markovian with finite memory l
METRO. Further assumptions will be made for the
validation study. The following quantities are needed for the setup and formal treatment of the
algorithm and are visualized in Figure 1 y figura 2:
Target process Y: a process of interest within the system (where Y = {Yt | t ∈ N}); el
choice of the target process is arbitrary and all the processes in the system can separately
be studied as targets.
Source processes X i: the remaining processes within the system (where i = 1, . . . , N − 1 y
X i = {Xi,t | t ∈ N}).
Sample number (or size) t: the number of samples recorded over time.
Replication number R: the number of replications of the same experiment (p.ej., ensayos).
Target present state Yt: the random variable (RV) representing the state of the target at time t
(where t ≤ T), whose information contributors will be inferred.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
t
/
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
t
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Candidate target past Y
C
j,S; Yt|Y
j,1; Yt|Y
2. Compute the maximum CMI value over candidates I∗
s
norte,s) for each
surrogate s = 1, . . . , S. Aquí, n denotes the number of candidates and hence the number
of comparisons. The obtained values I∗
1 , . . . , I∗
S provide the (empirical) null distribution
of the maximum statistic (ver tabla 1).
1,s, . . . , I(cid:5)
:= max(I(cid:5)
3. Calculate the p value for I∗
as the fraction of surrogate maximum statistic values that are
larger than I∗
I∗
.
4.
is deemed significant if the p value is smaller than αmax (es decir., the null hypothesis of
conditional independence for the candidate variable with the maximum CMI contribu-
tion is rejected at level αmax).
The variables and quantities used in the above algorithm are presented in Table 1. The key
goal in the surrogate generation is to preserve the temporal order of samples in the target time
series Yt (which is not shuffled) and preserve the distribution of the sources Cj while destroying
any potential relationships between the sources and the target (Vicente et al., 2011). This can
be achieved in multiple ways. If multiple replications (p.ej., ensayos) are available, surrogate data is
generated by shuffling the order of replications for the candidate Cj while keeping the order of
replications for the remaining variables intact. When the number of replications is not sufficient
to guarantee enough permutations, the embedded source samples within individual trials are
shuffled instead (see Chávez et al., 2003; Lizier et al., 2011; Verdes, 2005; Vicente et al., 2011;
and the summary by Lizier, 2014, Appendix A.5). Note that the generation of surrogates (steps
1-3) can be avoided when the null distributions can be derived analytically, Por ejemplo, con
Gaussian estimators (barnett & Bossomaier, 2012).
The same test is performed during the selection of the variables in the candidate sources
S
)
límite
límite
, . . . , In ≤ i
)
límite
Ij ≤ i
(cid:3)
límite
(cid:2)
PAG
= 1 −
= 1 − P (I1 ≤ i
norte
∏
j=1
= 1 − P (I1 ≤ i
= 1 − (1 − v
límite
)norte
FPR
)norte
(1)
(2)
Por lo tanto,
v
FPR
= 1 − (1 − αmax)1/norte
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
t
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
Curiosamente, Ecuación 2 shows that the maximum statistic correction is equivalent to the
Dunn-Šidák correction (Šidák, 1967). Performing a Taylor expansion of Equation 2 alrededor
αmax = 0 yields:
−
j−1
∏
k=0
(kn − 1)
j!
(cid:5)j
(cid:4) αtarget
norte
v
FPR
=
∞
∑
j=1
Truncating the Taylor series at j = 1 yields the first-order approximation
v
FPR
≈
αmax
norte
,
(3)
(4)
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
which coincides with the false positive rate resulting from the Bonferroni correction (Dickhaus,
2014). Además, since the summands in Equation 3 are positive for every j, the Taylor series
is lower bounded by any truncated series. En particular, the false positive rate resulting from
the Bonferroni correction is a lower bound for the v
FPR (the false positive rate for a single
variable resulting from the maximum statistic test), eso es, the maximum statistic correction is
less stringent than the Bonferroni correction.
Let us now study the effect of the maximum statistic test on the family-wise error rate t
FPR
for a single target while accounting for all the iterations performed during the step-down
Neurociencia en red
835
Large-scale network inference with multivariate transfer entropy
prueba, (es decir., t
Tenemos:
FPR is the probability that at least one of the selected sources is a false positive).
t
FPR
=
=
norte
∑
j=1
norte
∑
j=1
PAG(“the source selected on step j is false positive”)
(cid:7)
(cid:6)
αj
máximo
= αmax
1 − αn
máximo
1 − αmax
Por lo tanto,
t
FPR
≈ αmax
(5)
(6)
for the typical small values of αmax used in statistical testing (even in the limit of large n), cual
shows that αmax effectively controls the family-wise error rate for a single target.
The minimum statistic test is employed during the third main step of
Minimum statistic test.
the algorithm (pruning step) to remove the selected variables that have become redundant in
S
the context of the final set of selected source past variables X
recall was unchanged using S = 10, 000 for T = 100.) For longer time series (T = 10, 000),
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
t
/
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 3. Precision (arriba) and recall (abajo) for different network sizes, sample sizes, and dynamics. Left: Vector autoregressive process;
Right: Coupled logistic maps. Each subplot shows five curves, corresponding to different time series lengths (T = 100, 300, 1,000, 3,000,
10,000). The results for 10 simulations from different initial conditions are shown (low-opacity markers) in addition to the mean values (solid
markers). All the random networks have an average in-degree N p = 3.
Neurociencia en red
839
Large-scale network inference with multivariate transfer entropy
high performance according to all measures was achieved for both the VAR and CLM pro-
cesses, regardless of the size of the network. The high precision and specificity are due to
the effective control of the false positives, in accordance with the strict statistical significance
level αmax = 0.001 (the influence of αmax is further discussed in the following sections). El
inference algorithm was therefore conservative in the classification of the links.
Validation of False Positive Rate
The critical alpha level for statistical significance αmax is a parameter of the algorithm that is
designed to control the number of false positives in the network inference. As discussed in
the Statistical Tests section in the Methods, αmax controls the probability that a target is a false
positivo, eso es, that at least one of its sources is a false positive. This approach is in line with
the perspective that the goal of the network inference is to find the set of relevant sources for
each node.
A validation study was carried out to verify that the final number of false positives is consis-
tent with the desired level αmax after multiple statistical tests are performed. The false positive
rate was computed after performing the inference on empty networks, where every inferred
link is a false positive by definition (es decir., under the complete null hypothesis). The rate was
in good accordance with the critical alpha threshold αmax for all network sizes, as shown in
Cifra 4.
fMRI data from the Human Connectome Project resting-state dataset
The false positive rate validation was replicated in a scenario where the null hypothesis
(ver
held for real
Supporting Information). The findings are presented in the Supporting Information, together
with a note on autocorrelation. Notablemente, the results on fMRI data are in agreement with the
results on synthetic data shown in Figure 4.
False positive rate:
FP/(FP + TN).
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
t
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
t
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 4. Validation of false positive rate for a single target (t
FPR) on empty networks. The points
indicate the average false positive rate over 50 simulations of a vector autoregressive process
(T = 10,000). The horizontal marks indicate the corresponding 5th and 95th percentiles of the
expected range. These were computed empirically from the distribution of the random variable
(cid:10)Xj/N(cid:11), where Xj ∼ Binomial(norte, αmax) are i.i.d. random variables, and the angular brackets in-
dicate the finite average over 50 repetitions. The 5th percentile for N = 10 and N = 40 y
αmax = 10−3
are equal to zero and therefore omitted from the log-log plot. The identity function is
plotted as a reference (dashed line).
Neurociencia en red
840
Large-scale network inference with multivariate transfer entropy
Influence of Critical Level for Statistical Significance
Given the conservative results obtained for both the VAR and CLM processes (Cifra 3), a
natural question is to what extent the recall could be improved by increasing the critical alpha
level αmax and to what extent the precision would be negatively affected as a side effect.
In order to elucidate this trade-off, the analysis described above (Cifra 3) was repeated for
increasing values of αmax, with results shown in Figure 5. For the shortest time series (T = 100),
increasing αmax resulted in a higher recall and a lower precision, as expected; en el otro
mano, for the longest time series (T = 10,000), the performance measures were not significantly
affected. Curiosamente, for the intermediate case (T = 1,000), increasing αmax resulted in higher
recall without negatively affecting the precision.
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
t
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
t
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Cifra 5.
Influence of statistical significance threshold on network inference performance. Preci-
sion versus recall for different statistical significance levels (αmax = 0.05, 0.01, 0.001), correspond-
ing to different colors. The plots summarize the results for different dynamics (Top: Vector autore-
gressive process; Bottom: Coupled logistic maps), different time series lengths (T = 100, 1,000,
10,000), and different network sizes (norte = 10, 40, 70, 100, not distinguished). The arrows join
the mean population values for the lowest and highest significance levels, illustrating the average
trade-off between precision loss and recall gain.
Neurociencia en red
841
Large-scale network inference with multivariate transfer entropy
Inference of Coupling Lags
Hasta ahora, the performance evaluation focused on the identification the correct set of sources
for each target node, regardless of the coupling lags. Sin embargo, since the identification of the
correct coupling lags is particularly relevant in neuroscience (see Wibral et al., 2013, y
references therein), the performance of the algorithm in identifying the correct coupling lags
was additionally investigated.
Por construcción, a single coupling lag was imposed between each pair of processes (chosen
at random between one and five discrete time steps, as described in the Methods). el promedio
absolute error between the real and the inferred coupling lags was computed on the correctly
recalled sources and divided by the value expected at random (which is the average absolute
difference between two i.i.d. random integers in the [1, 5] interval). In line with the previous
results on precision, the absolute error on coupling lag is consistently much smaller than that
expected at random, even for the shortest time series (Cifra 6). Además, 1,000 muestras
were sufficient to achieve nearly optimal performance for both the VAR and the CLM processes,
regardless of the size of the network. Note that as T increases and the recall increases, the lag
error can increase (cf. T = 100 a 300 for the CLM process). This is perhaps because while the
larger T permits more weakly contributing sources to be identified, it is not large enough to
reduce the estimation error to make lag identification on these sources precise.
Estimators
Given its speed, the Gaussian estimator is often used for large datasets or as a first exploratory
step, even when the stationary distribution cannot be assumed to be Gaussian. The availability
of the ground truth allowed us to compare the performance of the Gaussian estimator and the
nearest-neighbor estimator on the nonlinear CLM process, which does not satisfy the Gaus-
sian assumption. As expected, the performance of the Gaussian estimator was lower than the
performance of the nearest-neighbor estimator for all network sizes (Cifra 7).
The hierarchical tests introduced in the Methods section allow running the network infer-
ence algorithm in parallel on a high-performance computing cluster. Such parallelization is
especially needed when employing the nearest-neighbor estimator. En particular, each target
node can be analyzed in parallel on a CPU (employing one or more cores) or a GPU, cual
is made possible by the CPU and GPU estimators provided by the IDTxl package (custom
Cifra 6. Average absolute error between the real and the inferred coupling lags, relative to the value expected at random. Results for
different dynamics (Left: Vector autoregressive process; Right: Coupled logistic maps), different time series lengths (T = 100, 300, 1,000,
3,000, 10,000), and different network sizes (norte = 10, 40, 70, 100). The error bars indicate the standard deviation over 10 simulations from
different initial conditions.
Neurociencia en red
842
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
t
/
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
Cifra 7. Gaussian versus nearest-neighbor estimator on the coupled logistic maps process. The precision (izquierda) and recall (bien) are plotted
against the network size and a fixed time series length (T = 10,000 muestras). The results for 10 simulations from different initial conditions
are shown (low-opacity markers) in addition to the mean values (solid markers). The statistical significance level αmax = 0.05 was employed;
an even larger gap between the recall of the estimators is obtained with αmax = 0.001.
OpenCL kernels were written for the GPU implementation). A summary of the CPU and GPU
run times is provided in the Supporting Information.
DISCUSIÓN
The algorithm presented in this paper provides robust statistical tests for network inference to
control the false positive rate. These tests are compatible with parallel computation on high-
performance computing clusters, which enabled the validation study on synthetic sparse net-
works of increasing size (10 a 100 nodos), using different dynamics (linear autoregressive
processes and nonlinear coupled logistic maps) and increasingly longer time series (100 a
10,000 muestras). Both the network size and the sample size are one order of magnitude larger
than previously demonstrated, showing feasibility for typical EEG and MEG experiments. El
results demonstrate that the statistical tests achieve the desired false positive rate and success-
fully address the multiple-comparison problems inherent in network inference tasks (Cifra 4).
The ability to control the false positives while building connectomes is a crucial prerequisite
for the application of complex network measures, to the extent that Zalesky et al. (2016) estafa-
cluded that “specificity is at least twice as important as sensitivity (es decir., recordar) when estimating
key properties of brain networks, including topological measures of network clustering, net-
work efficiency and network modularity.” The reason is that false positives occur more preva-
lently between network modules than within them, and the spurious intermodular connections
have a dramatic impact on network topology (Zalesky et al., 2016).
The trade-off between precision and recall when relaxing the statistical significance thresh-
old was further investigated (Cifra 5). When only 100 samples were used, the average recall
gain was more than five times smaller than the average precision loss. In our opinion, this result
is possibly due to the sparsity of the networks used in this study and suggests a conservative
choice of the threshold for sparse networks and short time series. The trade-off was reversed for
longer time series: cuando 1,000 samples were used, the average recall gain was more than five
times larger than the average precision loss. Finalmente, para 10,000 muestras, high precision and
recall were achieved (>98% on average) for both the vector autoregressive and the coupled
logistic maps processes, regardless of the statistical significance threshold.
For both types of dynamics, the network inference performance increased with the length of
the time series and decreased with the size of the network (Cifra 3). This is to be expected since
larger systems require more statistical tests and hence stricter conditions to control the family-
Neurociencia en red
843
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
/
t
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
t
.
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
wise error rate (false positives). Específicamente, larger networks result in wider null distributions
of the maximum statistic (es decir., larger variance), whereas longer time series have the opposite
efecto. Por lo tanto, for large networks and short time series, controlling the false positives can
have a negative impact on the ability to identify the true positives, particularly when the effect
tamaño (es decir., the transfer entropy value) is small.
Además, the superior ability of the nearest-neighbor estimator over the Gaussian estima-
tor in detecting nonlinear dependencies was quantified. There is a critical motivation for this
comparación: the general applicability of the nearest-neighbor estimators comes at the price
of higher computational complexity and a significantly longer run time, so that the Gaussian
estimator is often used for large datasets (or at least as a first exploratory step), even when the
Gaussian hypothesis is not justified. To investigate such a scenario, the Gaussian estimator was
tested on the nonlinear logistic map processes: while the resulting recall was significantly lower
than the nearest-neighbor estimator for all network sizes, it was nonetheless able to identify
over half of the links for a sufficiently large number (10,000) of time samples (Cifra 7).
The stationarity assumption about the time series corresponds to assuming a single regime
of neuronal activity in real brain recordings. If multiple regimes are recorded, which is typical
in experimental settings (p.ej., sequences of tasks or repeated presentation of stimuli interleaved
with resting time windows), different stationary regimes can be studied by performing the anal-
ysis within each time window. The networks obtained in different time windows can either
be studied separately and compared against each other or collectively interpreted as a single
evolving temporal network. To obtain a sufficient amount of observations per window, multiple
replications of the experiment under the same conditions are typically carried out. Replica-
tions can be assumed to be cyclo-stationary and estimation techniques exploiting this property
have been proposed (Gómez-Herrero et al., 2015; Wollstadt et al., 2014); these estimators are
also available in the IDTxl Python package. The convergence to the (unknown) causal net-
work was only proven under the hypotheses of stationarity, causal sufficiency, and the causal
Markov condition (Sun et al., 2015). Sin embargo, conditional independence holds under milder
suposiciones (Runge, 2018) and the absence of links is valid under general conditions. El
conditional independence relationships can, por lo tanto, be used to exclude variables in follow-
ing intervention-based causal experiments, making network inference methods valuable for
exploratory studies.
De hecho, the directed network is only one part of the model and provides the scaffold over
which the information-theoretic measures are computed. Por lo tanto, even if the structure of a
system is known and there is no need for network inference, information theory can still pro-
vide nontrivial insights on the distributed computation by modeling the information storage,
transfer, and modification within the system (Lizier, 2013). This decomposition of the predic-
tive information into the active information storage and transfer entropy components is one
out of many alternatives within the framework proposed by Chicharro & Ledberg (2012). Ar-
guably, the storage-transfer decomposition reflects the segregation-integration dichotomy that
has long characterized the interpretation of brain function (despreciar, 2010; Zeki & Shipp, 1988).
Information theory has the potential to provide a quantitative definition of these fundamental
but still unsettled concepts (Le et al., 2019). Además, information theory provides a new
way of testing fundamental computational theories in neuroscience, Por ejemplo, predictive
codificación (Brodski-Guerniero et al., 2017).
Tal como, information-theoretic methods should not be seen as opposed to model-based
approaches, but complementary to them (Friston et al., 2013). If certain physically motivated
Neurociencia en red
844
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
/
t
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
parametric models are assumed, the two approaches are equivalent for network inference:
maximizing the log-likelihood is asymptotically equivalent to maximizing the transfer entropy
(barnett & Bossomaier, 2012; Cliff et al., 2018). Además, different approaches can be com-
bined; Por ejemplo, the recent large-scale application of spectral DCM was made possible by
using functional connectivity models to place prior constraints on the parameter space (Razi
et al., 2017). Networks inferred using bivariate transfer entropy have also been employed to
reduce the model space prior to DCM analysis (Chan et al., 2017).
En conclusión, the continuous evolution and combination of methods show that network in-
ference from time series is an active field of research and there is a current trend of larger valida-
tion studies, statistical significance improvements, and reduction of computational complexity.
Information-theoretic approaches require efficient tools to employ nearest-neighbor estimators
on large datasets of continuous-valued time series, which are ubiquitous in large-scale brain
recordings (calcium imaging, EEG, MEG, resonancia magnética funcional). The algorithm presented in this paper is com-
patible with parallel computation on high-performance computing clusters, which enabled the
study of synthetic nonlinear systems of 100 nodes and 10,000 muestras. Both the network size
and the sample size are one order of magnitude larger than previously demonstrated, bring-
ing typical EEG and MEG experiments into scope for future information-theoretic network
inference studies. Además, the statistical tests presented in the Methods are generic and
compatible with any underlying conditional mutual information or transfer entropy estimators,
meaning that estimators applicable to spike trains (Spinney et al., 2017) can be used with this
algorithm in future studies.
EXPRESIONES DE GRATITUD
The authors acknowledge the Sydney Informatics Hub and the University of Sydney’s high-
performance computing cluster Artemis for providing the high-performance computing re-
sources that have contributed to the research results reported within this paper. Además,
the authors thank Aaron J. Gutknecht for commenting on a draft of this paper, and Oliver Cliff
for useful discussions and comments.
SUPPORTING INFORMATION
The network inference algorithm described in this paper is implemented in the open-source
Python software package IDTxl (Wollstadt et al., 2019), which is freely available on GitHub
(https://github.com/pwollstadt/IDTxl). en este documento, we refer to the current release (v1.0) en el
time of writing (doi:10.5281/zenodo.2554339).
The raw data used for the experiment presented in the Supporting Information (https://doi.
org/10.1162/netn_a_00092) is openly available on the MGH-USC Human Connectome Project
database (https://ida.loni.usc.edu/login.jsp).
ROLE INFORMATION
Leonardo Novelli: Conceptualización; Data Curation; Formal Analysis; Investigación; Soft-
mercancía; Validación; Visualización; Writing – Original Draft; Writing – Revisar & Editing. patricia
Wollstadt: Conceptualización; Software; Writing – Revisar & Editing. Pedro Mediano: Software;
Writing – Revisar & Editing. Michael Wibral: Conceptualización; Funding Acquisition; Método-
ology; Software; Supervisión; Writing – Revisar & Editing. Joseph T. Lizier: Conceptualización;
Funding Acquisition; Metodología; Software; Supervisión; Writing – Revisar & Editing.
Neurociencia en red
845
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
t
/
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
INFORMACIÓN DE FINANCIACIÓN
Joseph T. Lizier, Universities Australia/German Academic Exchange Service (DAAD) Australia-
Germany Joint Research Cooperation Scheme Grant: “Measuring Neural Information Synthe-
sis and Its Impairment,” Award Id: 57216857. Michael Wibral, Universities Australia/German
Academic Exchange Service (DAAD) Australia-Germany Joint Research Cooperation Scheme
Grant: “Measuring Neural Information Synthesis and Its Impairment,” Award Id: 57216857.
Joseph T. Lizier, Australian Research Council DECRA Grant, Award Id: DE160100630. Miguel
Wibral, Deutsche Forschungsgemeinschaft (DFG) Grant, Award Id: CRC 1193 C04. Joseph T.
Lizier, Australian Research Council Discovery Grant, Award Id: DP160102742.
REFERENCIAS
Atay, F. METRO., & Karabacak, Ö. (2006). Stability of coupled map net-
works with delays. SIAM Journal on Applied Dynamical Systems,
5(3), 508–527.
Ay, NORTE., & Polani, D. (2008). Information flows in causal networks.
Advances in Complex Systems, 11(01), 17–41.
barnett, l., Barrett, A. B., & Seth, A. k. (2009). Granger causality
and transfer entropy are equivalent for Gaussian variables. Phys-
ical Review Letters, 103(23), 238701.
barnett, l., & Bossomaier, t. (2012). Transfer entropy as a log-
likelihood ratio. Physical Review Letters, 109(13), 138105.
Benjamini, y., & Hochberg, Y. (1995). Controlling the false discov-
ery rate: a practical and powerful approach to multiple testing.
Journal of the Royal Statistical Society. Serie B (Methodological),
57(1), 289–300.
Bossomaier, T., barnett, l., Harré, METRO., & Lizier, j. t. (2016). An In-
troduction to Transfer Entropy. Springer International Publishing,
Chambridge, Reino Unido.
Brodski-Guerniero, A., Paasch, G.-F., Wollstadt, PAG., Özdemir, I.,
Lizier, j. T., & Wibral, METRO. (2017). Information-theoretic evidence
for predictive coding in the face-processing system. The Journal
of Neuroscience, 37(34), 8273–8283.
chan, j. S., Wibral, METRO., Wollstadt, PAG., Stawowsky, C., Brandl, METRO.,
Helbling, S., . . . Kaiser, j. (2017). Predictive coding over the
lifespan: Increased reliance on perceptual priors in older adults—
a magnetoencephalography and dynamic causal modelling
estudiar. bioRxiv Preprint, página 178095.
Chávez, METRO., Martinerie, J., & Le Van Quyen, METRO. (2003). Statistical
assessment of nonlinear causality: application to epileptic EEG
signals. Journal of Neuroscience Methods, 124(2), 113–128.
Chicharro, D., & Ledberg, A. (2012). Framework to study dynamic
dependencies in networks of interacting processes. Physical Re-
view E, 86(4), 041901.
Cliff, o., Prokopenko, METRO., & Fitch, R. (2018). Minimising the
Kullback-Leibler divergence for model selection in distributed
nonlinear systems. Entropy, 20(2), 51.
Cover, t. METRO., & tomás, j. A. (2005). Elements of Information The-
ory. John Wiley & Sons, Hoboken, Nueva Jersey, EE.UU, 2 edition.
Dickhaus, t. (2014). Simultaneous Statistical Inference. Saltador
Berlin Heidelberg, Berlina, Heidelberg.
Erd ˝os, PAG., & Rényi, A. (1959). On random graphs. Publicationes
Mathematicae Debrecen, 6, 290–297.
Faes, l., Nollo, GRAMO., & Porta, A. (2011). Information-based detec-
tion of nonlinear Granger causality in multivariate processes via
a nonuniform embedding technique. Physical Review E, 83(5),
051112.
Frenzel, S., & Pompe, B. (2007). Partial mutual information for cou-
pling analysis of multivariate time series. Physical Review Letters,
99(20), 204101.
Friston, k. j. (1994). Functional and effective connectivity in neu-
roimaging: a synthesis. Mapeo del cerebro humano, 2(1-2), 56–78.
Friston, k. J., Moran, r., & Seth, A. k. (2013). Analysing connectivity
with Granger causality and dynamic causal modelling. Actual
Opinion in Neurobiology, 23(2), 172–178.
Garland, J., James, R. GRAMO., & Bradley, mi. (2016). Leveraging infor-
mation storage to select forecast-optimal parameters for delay-
coordinate reconstructions. Physical Review E, 93(2), 022221.
Gómez-Herrero, GRAMO., Wu, w., Rutanen, K., Soriano, METRO., Pipa, GRAMO., &
Vicente, R. (2015). Assessing coupling dynamics from an ensem-
ble of time series. Entropy, 17(4), 1958–1970.
Granjero, C. W.. j. (1969). Investigating causal relations by econo-
metric models and cross-spectral methods. Econometrica, 37(3),
424–438.
kim, PAG., Rogers,
(2016). Causation
entropy identifies sparsity structure for parameter estimation
of dynamic systems. Journal of Computational and Nonlinear
Dynamics, 12(1), 011008.
J., & Bollt, mi. METRO.
J., Sol,
Kraskov, A., Stögbauer, h., & Grassberger, PAG. (2004). Estimating mu-
tual information. Physical Review E, 69(6), 066138.
li, METRO., Han, y., Una quemadura, METRO. J., romper la lanza, METRO., Poldrack, R. A., Brillar,
j. METRO., & Lizier, j. t. (2019). Transitions in brain-network level
information processing dynamics are driven by alterations in neu-
ral gain. bioRxiv Preprint, página 581538.
Lindner, METRO., Vicente, r., Priesemann, v., & Wibral, METRO. (2011).
TRENTOOL: A Matlab open source toolbox to analyse informa-
tion flow in time series data with transfer entropy. BMC Neuro-
ciencia, 12, 119.
Lizier, j. t. (2013). The Local Information Dynamics of Distributed
Computation in Complex Systems. Springer Berlin, Heidelberg.
Lizier, j. t. (2014). JIDT: An information-theoretic toolkit for study-
ing the dynamics of complex systems. Frontiers in robotics and
AI, 1, 11.
Lizier,
j. T., Heinzle,
J., Horstmann, A., Haynes,
J.-D., &
Prokopenko, METRO. (2011). Multivariate information-theoretic mea-
sures reveal directed information structure and task relevant
changes in fMRI connectivity. Journal of Computational Neuro-
ciencia, 30(1), 85–107.
Neurociencia en red
846
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
/
t
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
Large-scale network inference with multivariate transfer entropy
Lizier, j. T., Prokopenko, METRO., & Zomaya, A. Y. (2008). Local infor-
mation transfer as a spatiotemporal filter for complex systems.
Physical Review E, 77(2), 026110.
Lizier, j. T., Prokopenko, METRO., & Zomaya, A. Y. (2010). Información
modification and particle collisions in distributed computation.
Chaos, 20(3), 037109.
Lizier, j. T., Prokopenko, METRO., & Zomaya, A. Y. (2012). Local mea-
sures of information storage in complex distributed computation.
Information Sciences, 208, 39–54.
Lizier, j. T., & Rubinov, METRO. (2012). Multivariate construction of effec-
tive computational networks from observational data. Técnico
Report Preprint 25/2012, Max Planck Institute for Mathematics
in the Sciences.
Lorenz, H.-W. (1993). Chaotic dynamics in discrete-time economic
modelos. In Nonlinear Dynamical Economics and Chaotic Motion
(pages 119–166). Saltador, Berlina.
Montalto, A., Faes, l., & Marinazzo, D. (2014). MuTE: A MATLAB
toolbox to compare established and novel estimators of the
multivariate transfer entropy. PLoS ONE, 9(10), e109462.
Nichols, T., & Hayasaka, S. (2003). Controlling the familywise error
rate in functional neuroimaging: a comparative review. Statistical
Methods in Medical Research, 12(5), 419–446.
Paluš, METRO., Albrecht, v., & Dvoák, I. (1993). Information theoretic test
for nonlinearity in time series. Physics Letters A, 175(3-4), 203–209.
paninski, l. (2003). Estimation of entropy and mutual information.
Computación neuronal, 15(6), 1191–1253.
Razi, A., Seghier, METRO. l., zhou, y., McColgan, PAG., Zeidman, PAG.,
Parque, H.-J., despreciar, o., rees, GRAMO., & Friston, k. j. (2017). Large-
scale DCMs for resting-state fMRI. Neurociencia en red, 1(3),
222–241.
Roulston, METRO. S. (1999). Estimating the errors on measured entropy
información. Physica D: Nonlinear Phenomena,
and mutual
125(3-4), 285–294.
Rubinov, METRO., despreciar, o., van Leeuwen, C., & romper la lanza, METRO. (2009).
Symbiotic relationship between brain structure and dynamics.
BMC Neuroscience, 10, 55.
Runge, j. (2018). Causal network reconstruction from time se-
ries: from theoretical assumptions to practical estimation. Chaos,
28(7), 075310.
Runge, J., Heitzig, J., Petoukhov, v., & Kurths, j. (2012). Escaping
the curse of dimensionality in estimating multivariate transfer en-
tropy. Physical Review Letters, 108(25), 258701.
Runge, J., Nowack, PAG., Kretschmer, METRO., Flaxman, S., & Sejdinovic,
D. (2018). Detecting causal associations in large nonlinear time
series datasets. arXiv Preprint. arXiv: 1702.07007.
Schreiber, t. (2000). Measuring information transfer. Physical Re-
view Letters, 85(2), 461–464.
Schreiber, T., & Schmitz, A. (2000). Surrogate time series. Physica
D: Nonlinear Phenomena, 142(3-4), 346–382.
shannon, C. mi. (1948). A mathematical theory of communication.
Bell System Technical Journal, 27(3), 379–423.
Šidák, z. (1967). Rectangular confidence regions for the means of
multivariate normal distributions. Journal of the American Statis-
tical Association, 62(318), 626–633.
Sims, C. A. (1980). Macroeconomics and reality. Econometrica,
48(1), 1–48.
Spinney, R. MI., Prokopenko, METRO., & Lizier, j. t. (2017). Transfer en-
tropy in continuous time, with applications to jump and neural
spiking processes. Physical Review E, 95(3), 032319.
Spirtes, PAG., Glymour, C., & Scheines, R. (1993). Causation, Pre-
diction, and Search, volumen 81 of Lecture Notes in Statistics.
Springer New York.
despreciar, oh. (2010). Networks of the Brain. MIT Press Cambridge,
MAMÁ.
Strogatz, S. h. (2015). Nonlinear Dynamics and Chaos. CRC Press,
Boca Raton, Florida.
Sol, J., taylor, D., & Bollt, mi. METRO. (2015). Causal network inference by
optimal causation entropy. SIAM Journal on Applied Dynamical
Sistemas, 14(1), 73–106.
Takens, F. (1981). Detecting strange attractors in turbulence. En
Rand, D. and Young, l., editores, Dynamical Systems and Turbu-
lence, pages 366–381. Saltador, Berlina.
Vakorin, V. A., Krakovska, oh. A., & McIntosh, A. R. (2009). Estafa-
founding effects of indirect connections on causality estimation.
Journal of Neuroscience Methods, 184(1), 152–160.
Valdes-Sosa, PAG. A., Roebroeck, A., Daunizeau, J., & Friston, k. j.
(2011). Effective connectivity: influence, causality and biophysi-
cal modeling. NeuroImagen, 58(2), 339–361.
Vejmelka, METRO., & Paluš, METRO. (2008). Inferring the directionality of cou-
pling with conditional mutual information. Physical Review E,
77(2), 026214.
Verdes, PAG. F. (2005). Assessing causality from multivariate time se-
ries. Physical Review E, 72(2), 026222.
Vicente, r., Wibral, METRO., Lindner, METRO., & Pipa, GRAMO. (2011). Transferir
entropy—a model-free measure of effective connectivity for the
neurosciences. Journal of Computational Neuroscience, 30(1),
45–67.
Vlachos, I., & Kugiumtzis, D. (2010). Nonuniform state-space recon-
struction and coupling detection. Physical Review E, 82(1), 016207.
Wibral, METRO., Pampu, NORTE., Priesemann, v., Siebenhühner, F., Seiwert,
h., Lindner, METRO., . . . Vicente, R. (2013). Measuring information-
transfer delays. PLoS ONE, 8(2), e55809.
Wollstadt, PAG., Lizier, j. T., Vicente, r., Finn, C., Martínez-Zarzuela,
METRO., Mediano, PAG., . . . Wibral, METRO. (2019). IDTxl: The Information
Dynamics Toolkit xl: a Python package for the efficient analy-
sis of multivariate information dynamics in networks. Diario de
Open Source Software, 4(34), 1081.
Wollstadt, PAG., Martínez-Zarzuela, METRO., Vicente, r., Díaz-Pernas, F. J.,
& Wibral, METRO. (2014). Efficient transfer entropy analysis of non-
stationary neural time series. PLoS ONE, 9(7), e102833.
Brilla, A., Proporcionó, A., cocineros, l., Perdido, l. l., & romper la lanza,
(2014). Time-resolved resting-state brain networks. Pro-
the National Academy of Sciences, 111(28),
METRO.
cesiones de
10341–10346.
Brilla, A., Proporcionó, A., cocineros, l., Perdido, l. l., van den heuvel,
METRO. PAG., & romper la lanza, METRO. (2016). Connectome sensitivity or speci-
ficity: cuál es más importante? NeuroImagen, 142, 407–420.
Zeki, S., & Shipp, S. (1988). The functional logic of cortical connec-
ciones. Naturaleza, 335, 311–317.
Neurociencia en red
847
yo
D
oh
w
norte
oh
a
d
mi
d
F
r
oh
metro
h
t
t
pag
:
/
/
d
i
r
mi
C
t
.
metro
i
t
.
/
/
t
mi
d
tu
norte
mi
norte
a
r
t
i
C
mi
–
pag
d
yo
F
/
/
/
/
/
3
3
8
2
7
1
0
9
2
4
6
7
norte
mi
norte
_
a
_
0
0
0
9
2
pag
d
.
t
F
b
y
gramo
tu
mi
s
t
t
oh
norte
0
7
S
mi
pag
mi
metro
b
mi
r
2
0
2
3
