Abstract
Microbes form complex and dynamic ecosystems that play key roles in the health of the animals and plants with which they are associated. Such ecosystems are often represented by a directed, signed and weighted ecological network, where nodes represent microbial taxa and edges represent ecological interactions. Inferring the underlying ecological networks of microbial communities is a necessary step towards understanding their assembly rules and predicting their dynamical response to external stimuli. However, current methods for inferring such networks require assuming a particular population dynamics model, which is typically not known a priori. Moreover, those methods require fitting longitudinal abundance data, which is not readily available, and often does not contain the variation that is necessary for reliable inference. To overcome these limitations, here we develop a new method to map the ecological networks of microbial communities using steady-state data. Our method can qualitatively infer the inter-taxa interaction types or signs (positive, negative or neutral) without assuming any particular population dynamics model. Additionally, when the population dynamics is assumed to follow the classic Generalized Lotka-Volterra model, our method can quantitatively infer the inter-taxa interaction strengths and intrinsic growth rates. We systematically validate our method using simulated data, and then apply it to four experimental datasets of microbial communities. Our method offers a novel framework to infer microbial interactions and reconstruct ecological networks, and represents a key step towards reliable modeling of complex, real-world microbial communities, such as the human gut microbiota.
1. Introduction
The microbial communities established in animals, plants, soils, oceans, and virtually every ecological niche on Earth perform vital functions for maintaining the health of the associated ecosystems1-5. Recently, our knowledge of the organismal composition and metabolic functions of diverse microbial communities has markedly increased, due to advances in DNA sequencing and metagenomics6. However, our understanding of the underlying ecological networks of these diverse microbial communities lagged behind7. Mapping the structure of those ecological networks and developing ecosystem-wide dynamic models will be important for a variety of applications8, from predicting the outcome of community alterations and the effects of perturbations9, to the engineering of complex microbial communities7,10. We emphasize that the ecological network discussed here is a directed, signed and weighted graph, where nodes represent microbial taxa and edges represent direct ecological interactions (e.g., parasitism, commensalism, mutualism, amensalism or competition) between different taxa. This is fundamentally different from the correlation-based association or co-occurrence network7,11,12,13, which is undirected and does not encode any causal relations or direct ecological interactions, and hence cannot be used to faithfully predict the dynamic behaviour of microbial communities.
To date, existing methods for inferring the ecological networks of microbial communities are based on temporal abundance data, i.e., the abundance time series of each taxon in the microbial community14-19. The success of those methods has been impaired by at least one of the following two fundamental limitations. First, those inference methods typically require the a priori choice of a parameterized population dynamics model for the microbial community. These choices are hard to justify, given that microbial taxa in the microbial community interact via a multitude of different mechanisms7,20,21,22, producing complex dynamics even at the scale of two taxa23,24. Any deviation of the chosen model from the “true” model of the microbial community can lead to systematic inference errors, regardless of the inference method that is used19. Second, a successful temporal-data based inference requires sufficiently informative time-series data19,25. For many host-associated microbial communities, such as the human gut microbiota, the available temporal data are often poorly informative. This is due to the fact that such microbial communities often display stability and resilience26,27, which leads to measurements containing largely their steady-state behavior. For microbial communities such as the human gut microbiota, trying to improve the informativeness of temporal data is challenging and even ethically questionable, as it requires applying drastic and frequent perturbations to the microbial community, with unknown effects on the host.
To circumvent the above fundamental limitations of inference methods based on temporal data, here we developed a new method based on steady-state data, which does not require any external perturbations. The basic idea is as follows. Briefly, if we assume that the net ecological impact of species on each other is context-independent, then comparing equilibria (i.e., steady-state samples) consisting of different subsets of species would allow us to infer the interaction types. For example, if one steady-state sample differs from another only by addition of one species X, and adding X brings down the absolute abundance of Y, then we can conclude X inhibits the growth of Y. This very simple idea can actually be extended to more complicated cases where steady-state samples differ from each other by more than one species. Indeed, we rigorously proved that, if we collect enough independent steady states of the microbial community, it is possible to infer the microbial interaction types (positive, negative and neutral interactions) and the structure of the ecological network, without requiring any population dynamics model. We further derived a rigorous criterion to check if the steady-state data from a microbial community is consistent with the Generalized Lotka-Volterra (GLV) model15-19, a classic population dynamics model for microbial communities in human bodies, soils and lakes. We finally proved that, if the microbial community follows the GLV dynamics, then the steady-state data can be used to accurately infer the model parameters, i.e., inter-taxa interaction strengths and intrinsic growth rates. We validated our inference method using simulated data generated from various classic population dynamics models. Then we applied it to real data collected from four different microbial communities.
2. Results
Microbes do not exist in isolation but form complex ecological networks7. The ecological network of a microbial community is encoded in its population dynamics, which can be described by a set of ordinary differential equations (ODEs):
Here, fi(x(t)) ’s are some unspecified functions whose functional forms determine the structure of the underlying ecological network; is an N-dimensional vector with xi(t) denoting the absolute abundance of the i-th taxon at time t. In this work, we don’t require ‘taxon’ to have a particular taxonomic ranking, as long as the resulting abundance profiles are distinct enough across all the collected samples. Indeed, we can group microbes by species, genus, family or just operational taxonomic units (OTUs).
Note that in the right-hand side of Eq. (1) we explicitly factor out xi to emphasize that (i) without external perturbations those initially absent or later extinct taxa will never be present in the microbial community again as time goes by, which is a natural feature of population dynamics (in the absence of taxon invasion or migration); (ii) there is a trivial steady state where all taxa are absent; (iii) there are many non-trivial steady states with different taxa collections. We assume that the steady-state samples collected in a dataset correspond to those non-trivial steady states x∗ of Eq. (1), which satisfy , i = 1,…, N. For many host-associated microbial communities, e.g., the human gut microbiota, those cross-sectional samples collected from different individuals contain quite different collections of taxa (up to the taxonomic level of phylum binned from OTUs)26. We will show later that the number of independent steady-state samples is crucial for inferring the ecological network.
Mathematically, the intra- and inter-taxa ecological interactions (i.e., promotion, inhibition, or neutral) are encoded by the Jacobian matrix with matrix elements Jij(x(t)) = ∂fi(x(t))/∂xj. The condition Jij(x(t)) > 0 (< 0 or = 0) means that taxon j promotes (inhibits or doesn’t affect) the growth of taxon i, respectively. The diagonal terms Jii(x(t)) represent intra-taxa interactions. Note that Jij(x(t)) might depend on the abundance of many other taxa beyond i and j (due to the so-called “higher-order” interactions24,28-32).
The structure of the ecological network is represented by the zero-pattern of J(x(t)). Under a very mild assumption that holds if and only if Jij ≡ 0 (where xI and xk are two steady-state samples sharing taxon i), we find that the steady-state samples can be used to infer the zero-pattern of J(x(t)), i.e., the structure of the ecological network (see Supplementary Note 1.3 and 3 for details). Note that the network structure is interesting by itself and can be very useful in control theoretical analysis of microbial communities33. But in many cases, we are more interested in inferring the interaction types or strengths so that we can better predict the community’s response to perturbations.
The ecological interaction types are encoded in the sign-pattern of J(x(t)), denoted as sign(J(x(t))). To infer the interaction types, i.e., sign(J(x(t))), we make an explicit assumption that sign(J(x(t))) = const across all the observed steady-state samples. In other words, the nature of the ecological interactions between any two taxa does not vary across all the observed steady-state samples, though their interaction strengths might change. Note that the magnitude of Jij(x(t)) by definition may vary over different states, we just assume its sign remains invariant across all the observed samples/states. This assumption might be violated if those steady-state samples were collected from the microbial community under drastically different environmental conditions (e.g., nutrient availability34). In that case, inferring the interaction types becomes an ill-defined problem, since we have a “moving target” and different subsets of steady-state samples may offer totally different answers. Notably, as we will show later, the assumption is valid for many classic population dynamics models 35-39.
The assumption that sign(J(x(t))) = const can be falsified by analyzing steady-state samples. In Proposition 1 of Supplementary Note 1.4, we rigorously proved that if sign(J(x(t))) = const, then true multi-stability doesn’t exist. Equivalently, if a microbial community displays true multi-stability, then sign(J(x(t))) ≠ const. Here, a community of 0 taxa displays true multi-stability if there exists a subset of M (≤ N) taxa that has multiple different steady states, where all the M taxa have positive abundances and the other (N − M) taxa are absent. In practice, we can detect the presence of true multi-stability by examining the collected steady-state samples. If yes, then we know immediately that our assumption that sign(J(x(t))) = const is invalid and we should only infer the zero-pattern of J, i.e., the structure of the ecological network. If no, then at least our assumption is consistent with the collected steady-state samples, and we can use our method to infer sign(J(x(t))), i.e., the ecological interaction types. In short, by introducing a criterion to falsify our assumption, we significantly enhance the applicability of our method (see Supplementary Note 1.4 and Remark 6 for more detailed discussions).
Inferring interaction types
The assumption that sign(J(x(t))) = const enables us to mathematically prove that sign(J(x(t))) satisfies a strong constraint (Theorem 2 in Supplementary Note 1.4). By collecting enough independent steady-state samples, we can solve for the sign-pattern of J(x) and hence map the structure of the ecological network (Remarks 4 and 5 in Supplementary Note 1.4).
The basic idea is as follows. Let Ji be the set of all steady-state samples sharing taxon i. Then, for any two of those samples xI and xK, where the superscripts I,K ∈ Ji denote the collections of present taxa in those samples, we can prove that the sign-pattern of the i-th row of Jacobian matrix, denoted as a ternary vector si ∈ {−, 0, +}N, is orthogonal to (xI − xK) (Eq. (S3) in Supplementary Note 1.1). In other words, we can always find a real-valued vector , which has the same sign-pattern as si and satisfies yT · (xI − xK) = 0. If we compute the sign-patterns of all vectors orthogonal to (xI − xK) for all I, K ∈ Ji, then si must belong to the intersections of those sign-patterns, denoted as . In fact, as long as the number Ω of steady-state samples in is above certain threshold Ω*, then will contain only three sign-patterns {−a, 0, a} (Remark 5 in Supplementary Note 1.4). To decide which of these three remaining sign-patterns is the true one, we just need to know the sign of only one non-zero interaction. If such prior knowledge is unavailable, one can at least make a reasonable assumption that sii = ‘−’, i.e., the intra-taxa interaction Jii is negative (which is often required for community stability). When has more than three sign-patterns, we proved that the steady-state data is not informative enough in the sense that all sign-patterns in are consistent with the data available in (Remark 5 in Supplementary Note 1.4). This situation is not a limitation of any inference algorithm but of the data itself. To uniquely determine the sign-pattern in such a situation, one has to either collect more samples (thus increasing the informativeness of ) or use a priori knowledge of non-zero interactions.
We illustrate the application of the above method to small microbial communities with unspecified population dynamics (Fig. 1). For the two-taxa community (Fig. 1a), there are three possible types of equilibria, i.e., {x{1}, x{2}, x{1,2}}, depicted as colored pie charts in Fig. 1b. In order to infer s1 = (sign(J11), sign(J12)), we compute a straight line (shown in green in Fig. 1b) that is orthogonal to the vector (x{1,2} − x{1}) and passes through the origin. The regions (including the origin and two quadrants) crossed by this green line provide the set of possible sign-patterns that s1 may belong to. A priori knowing that J11 < 0, our method correctly concludes that s1 = (−, +). Note that J12 > 0 is consistent with the observation that with the presence of taxon 2, the steady-state abundance of taxon 1 increases (Fig. 1b), i.e., taxon 2 promotes the growth of taxon 1. We can apply the same method to infer the sign-pattern of s2 = (−, −).
For the three-taxa community (Fig. 1c), there are seven possible types of equilibria, i.e., {x{1}, x{2}, x{3}, x{1,2}, x{1,3}, x{2,3}, x{1,2,3}}. Four of them share taxon 1 (see colored pie charts in Fig. 1d). Six line segments connect the sample pairs, and represent vectors of the form (xI − xK), I, K ∈ J1 ={{1},{1,2},{1,3},{1,2,3}}. Considering a particular line segment (x{1,3} − x{1}), i.e., the solid blue line in Fig. 1d, we compute a plane (shown in orange in Fig. 1d) that is orthogonal to it and passes through the origin. The regions (including the origin and eight orthants) crossed by this orange plane provide a set of possible sign-patterns that s1 may belong to (see Fig. 1d). We repeat the same procedure for all other vectors (xI − xK), I, K ∈ J1, and compute the intersection of all the possible sign-patterns, finally yielding the minimum set to which s1 may belong to. If the sign of one non-zero interaction is known (J11 < 0 for this example), our method correctly infers the true sign-pattern s1 = (−,0,+). Repeating this process for samples sharing taxon 2 (or 3) will enable us to infer the sign-pattern s2 (or s3), respectively.
It is straightforward to generalize the above method to a microbial community of N taxa (see Supplementary Note 2.1 for details). But this brute-force method requires us to calculate all the sign-pattern candidates first, and then calculate their intersection to determine the minimum set that si will belong to. Since the solution space of sign-patterns is of size 3N, the time complexity of this brute force method is exponential with N, making it impractical for a microbial community with N > 10 taxa (Supplementary Note 2.2). To resolve this issue, we developed a heuristic algorithm that pre-calculates many intersection lines of (N − 1) non-parallel hyperplanes that pass through the origin and are orthogonal to (xI − xK), I, K ∈ ji. Based on these pre-calculated intersection lines, the algorithm determines using the most probable intersection line. The solution space of this heuristic algorithm is determined by the user-defined number of pre-calculated interaction lines (denoted as Ψ). Hence this algorithm naturally avoids searching the exponentially large solution space (see Supplementary Note 2.3 for details). Later on, we will show that this heuristic algorithm can indeed infer the interaction types with high accuracy.
In reality, due to measurement noise and/or transient behavior of the microbial community, the abundance profiles of the collected samples may not exactly represent steady states of the microbial community. Hence for certain Jij ’s their inferred signs might be wrong. Using simulated data, we will show later that for considerable noise level the inference accuracy is still reasonably high.
Inferring interaction strengths
To quantitatively infer the inter-taxa interaction strengths, it is necessary to choose a priori a parameterized dynamic model for the microbial community. The classical GLV model can be obtained from Eq. (1) by choosing where is the intrinsic growth rate vector and is the interaction matrix characterizing the intra- and inter-taxa interactions.
From Eq. (2) we can easily calculate the Jacobian matrix J, which is nothing but the interaction matrix A itself. This also reflects the fact that the value of aij quantifies the interaction strength of taxon j on taxon i. The GLV model considerably simplifies the inference of the ecological network, because we can prove that ai · (x1 − xK) = 0, for all I, K ∈ Ji, where ai ≡ (ai1,…, aiN) represents the i-th row of A matrix (Supplementary Note 5.2). In other words, all steady-state samples containing the i-th taxon will align exactly onto a hyperplane, whose orthogonal vector is parallel to the vector ai that we aim to infer (Fig. 2a, Theorem 3 of Supplementary Note 5.1). Thus, for the GLV model, the inference from steady-state data reduces to finding an (N − 1)-dimensional hyperplane that “best fits” the steady-state sample points {xI|I ∈ Ji} in the N-dimensional state space. In order to exactly infer ai, it is necessary to know the value of at least one non-zero element in ai, say, aii. Otherwise, we can just determine the relative interaction strengths by expressing aij in terms of aii. Once we obtain ai, the intrinsic growth rate ri of the i-th taxon can be calculated by averaging (−ai · xI) over all I ∈ Ji, i.e., all the steady-state samples containing taxon i.
In case the samples are not collected exactly at steady states of the microbial community or there is noise in abundance measurements, those samples containing taxon i will not exactly align onto a hyperplane. A naive solution is to find a hyperplane that minimizes its distance to those noisy samples. But this solution is prone to induce false positive errors and will yield non-sparse solutions (corresponding to very dense ecological networks). This issue can be partly alleviated by introducing a Lasso regularization40, implicitly assuming that the interaction matrix A in the GLV model is sparse. However, the classical Lasso regularization may induce a high false discovery rate (FDR), meaning that many zero interactions are inferred as non-zeros ones. To overcome this drawback, we applied the Knockoff filter41 procedure, allowing us to control the FDR below a desired user-defined level q > 0 (see Supplementary Note 5.3 for details).
The observation that for the GLV model all noiseless steady-state samples containing the i-th taxon align exactly onto a hyperplane can also be used to characterize how much the dynamics of the i-th taxon in a real microbial community deviates from the GLV model. This deviation can be quantified by the coefficient of determination (denoted by R2) of the multiple linear regression when fitting the hyperplane using the steady-state samples (Fig. 2b). If R2 is close to 1 (the samples indeed align to a hyperplane), we conclude that the dynamics of the microbial community is consistent with the GLV model, and hence the inferred interaction strengths and intrinsic growth rates are reasonable. Otherwise, we should only aim to qualitatively infer the ecological interaction types that do not require specifying any population dynamics.
Validation on simulated data
Interaction types
To validate the efficacy of our method in inferring ecological interaction types, we numerically calculated the steady states of a small microbial community with N = 8 taxa, using four different population dynamics models35-39: Generalized Lotka-Volterra (GLV), Holling Type II (Holling II), DeAngelis-Beddington (DB) and Crowley-Martin (CM) models (see Supplementary Note 4 for details). Note that all these models satisfy the requirement that the sign-pattern of the Jacobian matrix is time-invariant. To infer the ecological interaction types among the 8 taxa, we employed both the brute-force algorithm (with solution space ~ 38 = 6,561) and the heuristic algorithm (with solution space given by the number of the pre-calculated intersections chosen as Ψ = 5N = 40).
In the noiseless case, we find that when the number of steady-state samples satisfies Ω > 3N, the heuristic algorithm outperformed the brute-force algorithm for datasets generated from all the four different population dynamics models (Fig. 3a). This result is partly due to the fact that the former requires much fewer samples than the latter to reach high accuracy (the percentage of correctly inferred interaction types). However, when the sample size Ω is small (< 3N), the heuristic algorithm completely fails while the brute-force algorithm still works to some extent.
We then fix Ω = 5N, and compare the performance of the brute-force and heuristic algorithms in the presence of noise (Fig. 3b). We add artificial noise to each non-zero entry of a steady-state sample x1 by replacing with , where is a random number uniformly distributed in the interval and η ≥ 0 quantifies the noise level. We again find that the heuristic algorithm works better than the brute-force algorithm for datasets generated from all the four different population dynamics models.
The above encouraging results on the heuristic algorithm prompt us to systematically study the key factor to obtain an accurate inference, i.e., the minimal sample size Ω* (Fig. 3c, d). Note that for a microbial community of N taxa, if we assume that for any subset of the N taxa there is only one stable steady state such that all the corresponding taxa have non-zero abundance, then there are at most Ωmax = (2N − 1) possible steady-state samples (Of course, not all of them will be ecologically feasible. For example, certain pair of taxa will never coexist.) In general, it is unnecessary to collect all possible steady-state samples to obtain a highly accurate inference result. Instead, we can rely on a subset of them. To demonstrate this, we numerically calculated the minimal sample size Ω* we need to achieve a highly accurate inference of interaction types. We considered two different taxa presence patterns: (1) uniform: all taxa have equal probability of being present in the steady-state samples (inset of Fig. 3c); and (2) heterogeneous: a few taxa have higher presence probability than others, reminiscent of human gut microbiome samples26 (inset of Fig. 3d). We found that for the steady-state data generated from all the four population dynamics models, Ω* always scales linearly with N in both taxa presence patterns, and the uniform taxa presence pattern requires much fewer samples (Fig. 3c,d).
Note that as N grows, the total possible steady-state samples Ωmax increases exponentially, while the minimal sample size Ω* we need for high inference accuracy increase linearly. Hence, interestingly, we have Ω*/Ωmax → 0 as N increases. This suggests that as the number of taxa increases, the proportion of samples needed for accurate inference actually decreases. This is a rather counter-intuitive result because, instead of a “curse of dimensionality”, it suggests that a “blessing of dimensionality” exists when using the heuristic algorithm to infer interaction types for microbial communities with a large number of taxa.
Interaction strengths
To validate our method in quantitatively inferring inter-taxa interaction strengths, we numerically calculated steady states for a microbial community of N = 50 taxa, using the GLV model with aii = − 1 for all taxa.
In the noiseless case, if during the inference we know exactly aii = −1 for all taxa, then we can perfectly infer the inter-taxa interaction strengths aij’s and the intrinsic growth rates ri’s (see Fig. 4a). To study the minimal sample size Ω* required for perfect inference in the noiseless case, we again consider two different taxa presence patterns: (1) uniform; (2) heterogeneous. We find that for both taxa presence patterns Ω* scales linearly with N, though the uniform taxa presence pattern requires much fewer samples (Fig. 4b).
In the presence of noise, and if we don’t know the exact values of aii’s, but just assume they follow a half-normal distribution , we can still infer aij’s and ri’s with reasonable accuracy (with the normalized root-mean-square error NRMSE < 0.08), for noise level η < 0.3 (Fig. 4c-f). However, we point out that the classical Lasso regularization could induce many false positive, and the false discovery rate (FDR) reaches 0.448 at noise level η = 0.1, indicating that almost half of inferred non-zero interactions are actually zero (Fig. 4c). Indeed, even with a noise level η = 0.04, the classical Lasso already yields FDR~0.45, staying there for higher η (Fig. 4d).
In many cases, we are more concerned about low FDR than high false negative rates, because the topology of an inferred ecological network with even many missing links can still be very useful in the study of its dynamical and control properties42. To control FDR below a certain desired level q = 0.2, we applied the Knockoff filter41 (Fig. 4e), finding that though it will introduce more false negatives (see the horizontal bar in Fig. 4e), it can control the FDR below 0.2 for a wide range of noise level (Fig. 4f).
We also found that applying this GLV inference method to samples obtained from a microbial community with non-GLV dynamics leads to significant inference errors even in the absence of noise (Supplementary Fig. 9).
Application to experimental data
A synthetic soil microbial community of eight bacterial species43
This dataset consists of steady states of a total of 101 different species combinations: all 8 solos, 28 duos, 56 trios, all 8 septets, and 1 octet (see Supplementary Note 6.1 for details). For those steady-state samples that started from the same species collection but with different initial conditions, we average over their final steady states to get a representative steady state for this particular species combination.
In the experiments, it was found that several species grew to a higher density in the presence of an additional species than in monoculture. The impact of each additional species (competitor) j on each focal species i can be quantified by the so-called relative yield, defined as: , which represents a proxy of the ground truth of the interaction strength that species j impacts species i. A negative relative yield indicates growth hindrance of species j on i, whereas positive values indicated facilitation (Fig. 5a). Though quantifying the relative yield is conceptually easy and implementable for certain small microbial communities (see Supplementary Note 7 for details), for many host-associated microbial communities with many taxa, such as the human gut microbiota, measuring these one- and two-species samples is simply impossible. This actually motivates the inference method we developed here.
Before we apply our inference method, to be fair we remove all those steady states involving one- or two species, and analyse only the remaining 65 steady states. (Note that for N = 8, the number of total possible steady states is Ωmax = 255. Hence we only use roughly one quarter of the total possible steady states.) During the inference, we first check if the population dynamics of this microbial community can be well described by the GLV model. We find that all the fitted hyperplanes show small R2, indicating that the GLV model is not appropriate to describe the dynamics of this microbial community (Supplementary Fig. S10b). Hence, we have to aim for inferring the ecological interaction types, without assuming any specific population dynamics model.
Since this microbial community has only eight species, we can use the brute-force algorithm to infer the sign-pattern of the 8×8 Jacobian matrix, i.e., the ecological interaction types between the 8 species (The results of using a heuristic algorithm are similar and described in the Supplementary Fig. 10c). Compared with the ground truth obtained from the relative yield (Fig. 5a), we find that 50 (78.13%) of the 64 signs were correctly inferred, 10 (15.62%) signs were falsely inferred (denoted as ‘×’), and 4 (6.25%) signs cannot be determined (denoted as ‘?’) with the information provided by the 65 steady states (Fig. 5b).
We notice that the relative yield of many falsely inferred interactions is weak (with the exception of REa,Pch and REa,Pf). We conjecture that these errors are caused by noise or measurement errors in the experiments. To test this conjecture, we analyzed the robustness of each inferred sij by calculating the percentage of unchanged sij after adding perturbations to the samples (Fig. 5c). Similar to adding noise to simulated steady-state data, here we add noise to each non-zero entry of a sample xI such that where . The more robust the inferred results are, the higher the percentage of unchanged signs as η is increased. We found that most of the inferred signs were robust: the percentage of unchanged signs remained nearly 80% up to noise level η = 0.3 (Fig. 5c). Specifically, Fig. 5d plots the percentage of unchanged signs of the inferred Jacobian matrix when η = 0.04. We found that even if the perturbation is very small, 5 of the 10 falsely inferred sij in Fig. 5b changed their signs very frequently (blue entries with label ‘×’ in Fig. 5d). In other words, those interactions were very sensitive to noise, suggesting that some falsely inferred signs in Fig. 5b were largely caused by the noise.
A synthetic bacterial community of maize roots44
There are 7 bacterial species (Ecl, Sma, Cpu, Opi, Ppu, Hfr and Cin) in this community. This dataset consists of in total 8 steady-state samples: 7 sextets and 1 septet. We verified that this community cannot be described by the GLV dynamics (Supplementary Fig. 11).
Using only the 7 sextets (i.e., 7 steady-state samples involving 6 of the 7 species), we inferred the sign-pattern of the Jacobian matrix (Fig. 6a). Based on the sign of Jij, we can predict how the abundance of species-i in a microbial community will change, when we add species-j to the community. For example, if we add Ecl to a community consisting of the other 6 species (i.e., Sma, Cpu, Opi, Ppu, Hfr and Cin), we predict that the abundance of Sma, Opi, Ppu, Hfr and Cin will increase, while the abundance of Cpu will decrease (first column of Fig. 6b). Note that our prediction only considers the direct ecological interactions between species and ignores the indirect impact among species. Indeed, Ecl promotes Opi, but Ecl also promotes Hfr that inhibits Opi. Hence the net effect of Ecl on Opi is hard to tell without knowing the interaction strengths. Nevertheless, we found that our prediction is consistent with experimental observation (Fig. 6b, first column).
We then systematically compared our predictions of species abundance changes with experimental observations. There are in total 7 sextets, corresponding to the 7 columns in Fig. 6b. We add the corresponding missing species back to the community, and check the abundance changes of the existing 6 species. There are in total 6×7 = 42 abundance changes. We found that our inferred sign-pattern of the Jacobian matrix (Fig. 6a) can correctly predict 30 of the 42 abundance changes (accuracy ~71.43%). Moreover, for those false predictions, the detailed values of the abundance changes are actually relatively small (comparing to those of correct predictions). Note that we only used 7 steady-samples to infer the interaction types. If we have more steady-state samples available, we assume the prediction accuracy of our method can be further improved.
In the Supplementary Notes 6.3, 6.4, we also demonstrated the application of our method to two additional datasets.
3. Discussion
In this work, we developed a new inference method to map the ecological networks of microbial communities using steady-state data. Our method can qualitatively infer ecological interaction types (signs) without specifying any population dynamics model. Furthermore, we show that steady-state data can be used to test if the dynamics of a microbial community can be well described by the classic GLV model. When GLV is found to be adequate, our method can quantitatively infer inter-taxa interaction strengths and the intrinsic growth rates.
The proposed method bears some resemblance to previous network reconstruction methods based on steady-state data45. But we emphasize that, unlike the previous methods, our method does not require any perturbations applied to the system nor sufficiently close steady states. For certain microbial communities such as the human gut microbiota, applying perturbations may raise severe ethical and logistical concerns.
Note that our method requires the measurement of steady-state samples and absolute taxon abundances. For systems that are in frequent flux, where steady-state samples are hard to collect, our method is not applicable. Moreover, it fails on analyzing the relative abundance data (see Supplementary Note 2.4 for details). Note that the compositionality of relative abundance profiles also represents a major challenge for inference methods based on temporal data15,19. Fortunately, for certain small laboratory-based microbial communities, we can measure the absolute taxon abundances in a variety of ways, e.g., selective plating46, quantitative polymerase chain reaction (qPCR)15,16,47,48, flow cytometry49, and fluorescence in situ hybridization (FISH)50. For example, in the study of a synthetic soil microbial community of eight bacterial species43, the total cell density was assessed by measuring the optical density and species fractions (relative abundance) were determined by plating on nutrient agar plates. In recent experiments evaluating the dynamics of Clostridium difficile infection in mice models15,16, two sources of information were combined to measure absolute abundances: (1) data measuring relative abundances of microbes, typically consisting of counts (e.g., high-throughput 16S rRNA sequencing data); and (2) data measuring overall microbial biomass in the sample (e.g., universal 16S rRNA qPCR).
In contrast to the difficulties encountered in attempts to enhance the informativeness of temporal data that are often used to infer ecological networks of microbial communities, the informativeness of independent steady-state data can be enhanced by simply collecting more steady-state samples with distinct taxa collection (For host-associated microbial communities, this can be achieved by collecting steady-state samples from different hosts). Our numerical analysis suggests that the minimal number of samples with distinct taxa collections required for robust inference scales linearly with the taxon richness of the microbial community. Our analysis of experimental data from a small synthetic microbial community of eight species shows that collecting roughly one quarter of the total possible samples is enough to obtain a reasonably accurate inference. Furthermore, our numerical results suggest that this proportion can be significantly lower for larger microbial communities.
This blessing of dimensionality suggests that our method holds great promise for inferring the ecological networks of large and complex microbial communities, such as the human gut microbiota. There are two more encouraging facts that support this idea. First of all, it has been shown that the composition of the human gut microbiome remains stable for months and possibly even years until a major perturbation occurs through either antibiotic administration or drastic dietary changes51-54. The striking stability and resilience of human gut microbiota suggest that the collected samples very likely represent the steady states of the gut microbial ecosystem. Second, for healthy adults the gut microbiota displays remarkable universal ecological dynamics55 across different individuals. This universality of ecological dynamics suggests that microbial abundance profiles of steady-state samples collected from different healthy individuals can be roughly considered as steady states of a conserved “universal gut dynamical” ecosystem and hence can be used to infer its underlying ecological network. Despite the encouraging facts, we emphasize that there are still many challenges in applying our method to infer the ecological network of the human gut microbiota. For example, the assumption of invariant ecological interaction types (i.e., promotion, inhibition, or neutral) between any two taxa needs to be carefully verified. Moreover, our method requires the measurement of absolute abundances of taxa.
We expect that additional insights into microbial ecosystems will emerge from a comprehensive understanding of their ecological networks. Indeed, inferring ecological networks using the method developed here will enable enhanced investigation of the stability56 and assembly rules57 of microbial communities as well as facilitate the design of personalized microbe-based cocktails to treat diseases related to microbial dysbiosis9,10.
Contributions
Y.-Y.L conceived the project. Y.-Y.L and M.T.A. designed the project. Y.X. and M.T.A. did the analytical calculations. Y.X. did the numerical simulations and analyzed the empirical data. All authors analyzed the results. Y.-Y.L., Y.X. and M.T.A. wrote the manuscript. All authors edited the manuscript.
Author Information
The authors declare no competing financial interests.
2. BRUTE-FORCE AND HEORISTIC ALGORITHMS
Here we introduce the methodology for inferring the zero- or sign-patterns of the Jacobian matrix associated with the population dynamics of a microbial community. In essence, the inference of the zero-pattern is similar to the inference of the sign-pattern. Indeed, the only difference is that the former doesn’t care if the non-zero values are positive or negative. This implies that the complexity for inferring the network topology and interaction types are roughly the same. Here, for simplicity we describe the algorithms for inferring sign-patterns. All the algorithms (and pseudo codes) can be easily modified to infer the zero-pattern.
2.1. Brute-force algorithm
Theorems 1 and 2, together with Remarks 3 and 5 can be used to construct an algorithm to obtain all admissible sign-patterns for given steady-state data. Indeed, by enumerating all possible sign-patterns, we can use the liner program in Eq. (S5) to check if each of the possible 3N sign-patterns is admissible for taxon i, see Algorithm 1.
A brute-force algorithm to compute
Input: The collection of matrices Mi, being the difference between all two samples containing species is the nomber of samples Output: The sign-pattern set of 1: ← Enomeration of all the possible combinations {−, 0, +}N 2: for each j-th row in Mi do 3: 4: for each kth-subset in do 5: if find subject to v⊤Mi[j,:] = 0 and then 6: 7: end if 8: end for 9: 10: end for
Below, we illustrate the application of the brute-force algorithm for a microbial community with N = 3 taxa.
Here we consider the case of a microbial community with N = 3 taxa and population dynamics given by the so-called Crowley-Martin functional response [5]. The ODEs are:
We set the parameters r1 = 1, r2 = 5, r3 = 1.5, h = 0.2 and
Notice again that sign(J) = sign(A). We focus on reconstructing sign(J1), as the same procedure applies to the other taxa. With the given parameters, the feasible steady states of Eq. (S7) where taxon 1 is present are: constituting the available steady-state samples for taxon 1. We apply Algorithm 1 to this dataset obtaining
Note that . Providing for example sign(J11) < 0 as prior information, we correctly infer that sign(J1) = (−, 0, +).
2.2. Computational complexity of the brute-force algorithm
Algorithm 1 strongly relies on the enumeration of all 3N possible sign-patterns in , since it needs to test if each one of them is admissible for the given data. If the set has ni elements, there will be ni(ni − 1)/2 vectors of the form xI − xK with . According to Algorithm 1, for each of those vectors and each of the possible 3N sign-patterns, we will need to run the linear program (S5) to check if there is an orthogonal vector with the desired sign-pattern.
If we assume that the linear program can be solved with N operations, then, for each taxon, Algorithm 1 requires to perform a number of operations in the order of
If we have more than two samples (i.e., ni > 2), then ni(ni − 1)/2 > 1 and consequently C > N3N. Hence, for 100 taxa, we will need to perform at least 5.19 × 1049 operations for the reconstruction of each taxon —which is a number with the same order of magnitude as the number of atoms in Earth. Further, the linear programming used in the brute-force method can also be time consuming even for a small microbial community with N ~ 10. Consequently, applying the enumeration procedure is only reasonable for a community with N ~ 10, since in this case only around 106 operations are needed to infer the sign-pattern of the Jacobian corresponding to each taxon.
The above two limitations motivated us to develop a more efficient reconstruction method. This method has two main ingredients. First, a graph-based approach to quickly check whether a region can be crossed by a hyperplane, circumventing the need to solve the linear program. Second, an heuristic algorithm efficiently explore the solution space and to infer the ecological interaction types.
2.3. Inference using the heuristic algorithm
In practice, for large microbial communities with unknown dynamics, the inference of ecological interactions according to Algorithm 1 has two major drawbacks:
Checking if an orthant is crossed by a given hyperplane using the linear program of Eq. (S5) is computationally expensive.
The number of orthants that is necessary to check (i.e., the solution space) increases as 3N, that is, exponentially in the number of taxa.
To circumvent the first drawback, we introduce an alternative method based on the notion of sign-satisfaction. To address the second challenge, we propose an heuristic algorithm with user-defined time complexity to infer the sign-pattern of Ji.
2.3.1. Formulating the sign-satisfaction problem
Consider a real-valued vector . Then, solving the linear program Eq. (S5) is equivalent to solving the following sign-satisfaction problem:
Notice that from a geometrical viewpoint, solving Eq. (S8) is just finding the orthants of crossed by the hyperplane orthogonal to (xI − xK).
In the next example, we illustrate how the sign-satisfaction formulation allows us to quickly discard orthants of that cannot be crossed by such hyperplane:
In vector form, Eq. (S3) in Example 5 can be written as if we take two samples sharing taxon 1. Thus, the sign-satisfaction for Example 5 can be written as
Note that, for example, the choice sign(y) = (−, 0, −) cannot satisfy the above condition regardless of the particular value of y, because the inner product is the sum of two positive numbers, which can never be zero.
A systematic method to extend the above example and solve the sign-satisfaction problem is discussed next.
2.3.2. A graph-based approach to solving the sign-satisfaction problem
We illustrate the basic idea using a small example, and then discuss the general case.
In Example 6, the sign-satisfaction problem required that where sign(y1,y2,y3) = sign(J11,J12,J13). We map the above equation to the sign-satisfaction graph in Supplementary Fig. 4, where each element of J1 corresponds to a column and each element of sign(J1) has three possibilities (i.e., ‘−’,‘0’ or ‘+’). Each node in Supplementary Fig. 4 is divided in two parts: the left is an entry of sign(xI − xK) and the right is an entry of sign(J1). The color of each node encodes the sign of the product of left and right parts: grey is zero, red is positive and blue is negative. Next we introduce edges starting from each node and pointing to all nodes located in the next column to its right. With this formulation, the solutions to the sign-satisfaction problem (S8) reduces to finding the paths in the sign-satisfaction graph that satisfy one of the following two conditions:
the path contains red (representing positive values) and blue (representing negative values) nodes simultaneously, or
the path contains only gray (representing zero values) nodes.
The above two conditions guarantee that the sum of the product of sign(Jij) and (xI − xK) can be zero. At this step, it is also useful to introduce the prior information that is available, such as Jii < 0, allowing us to collapse columns of nodes in the sign-satisfaction graph to single nodes (Supplementary Fig. 4).
In a general case, for a given (xI − xK), the construction of the sign-satisfaction graph is as follows:
the graph consists of N columns, with each column having three nodes;
each node in the graph is divided into two parts: the left correspond to an entry of sign(xI − xK) and the right to an entry of sign(Jj);
each node is colored according to the sign of the product of the left and right parts: zero is grey, positive is red and negative is blue;
directed edges are included from a node to all the nodes in next column.
Finally, a solution to the sign-satisfaction problem of Eq. (S8) corresponds to a path from the first column to N-th column satisfying either condition (i) or (ii) listed above. In such case, a possible sign-pattern of J1 consists of the sign in the right part of each node in the path. For instance, the paths with yellow directed edges in Supplementary Fig. 4 correspond to the possible sign-pattern of J1, i.e., (−,+,+), (−, −, +) and (−, 0, +).
By using the sign-satisfaction graph, it is very efficient to test if the hyperplane orthogonal to (x1 − xK) crosses some orthants of , because it reduces to checking if its corresponding vector in {−, 0, +}N satisfies either condition (i) or (ii). However, finding all orthants crossed by such orthogonal hyperplane remains challenging, since the sign-satisfaction graph did not decrease the dimension of the solution space (that remains with exponential size 3N). To address this issue, next we introduce a method to efficiently sample paths in the sign-satisfaction graph.
2.3.3. Use the intersection of hyperplanes to sample paths in the sign-satisfaction graph
As discussed before, with the sign-satisfaction graph the solution space is still exponential (with size 3N−1, where the term N − 1 comes from assuming we know that Jii < 0 as prior information). One possibility to circumvent this problem would be to randomly sample paths in the sign-satisfaction graph and check if they satisfy conditions (i) or (ii). This would not work, however, since the probability of sampling the true “sign(Ji)” is only X/3N−1 − where X is the number of sampled paths − and this probability approaches zero as N increases. To alleviate this problem, next we propose a method to sample paths in the sign-satisfaction graph with certain preference.
This method has four steps and depends on an user-defined parameter Ψ > 1 specifying the number of times the procedure is repeated:
Construct the matrix of the difference of all the sample pairs. Consider the set of all vectors . Let be a matrix constructed by stacking all the vectors, where is the number of samples containing taxon i. By construction, each column of Mi is the normal vector of a hyperplane orthogonal to the difference of the corresponding sample pair.
Randomly sample (N − 1) hyperplanes. Choose randomly N − 1 columns from Mi.
Find the intersection of the (N − 1) sampled hyperplanes to obtain an intersection line. This can be done by finding the kernel of the matrix obtained by stacking the chosen columns. Note that the randomly sampled (N − 1) hyperplanes not always intersect in a line, because some hyperplanes might be parallel. However, this situation is non-generic in . Thus, if the randomly sampled hyperplanes do no intersect as a line, we return to step 2 and choose a new subset of columns.
Count how many hyperplanes cross the region of the intersection line using the sign-satisfaction graph. The sign-pattern of this intersection line represent the three orthants in crossed by all those (N − 1) hyperplanes. For the remaining hyperplanes in Mi (i.e., the rest of the columns in Mi), let be the number of those hyperplanes that cross these three orthants. We normalize using , so that ϕ ∈ [0,1]. Notice that ϕ = 1 means that this sign-pattern of the intersection line meets the requirements of sign-satisfaction for all the sample pairs. Therefore, the magnitude of the computed ϕ can be seen as the confidence of this potential solution to be a solution of the sign-satisfaction problem.
Go back to Step (2) until Ψ > 1 intersection lines have been computed.
In summary, selecting the intersection line can be seen as a “preference” sampling in the sign-satisfaction graph, because this intersection line can be crossed by at least (N − 1) hyperplanes in Mi.
We illustrate the basic idea of the above discussion in the following example:
We compute the difference vector of all the sample pairs (the samples contains taxon 1) in Example 5 and stack them in the following matrix:
Each column of M1 is the difference of a sample pair, corresponding to the normal vector of a plane orthogonal to the associated (xI − xK). In Supplementary Fig. 5a, the intersection line (black line) is intersected by the planes where each of normal vectors respectively corresponds to the 1-st and 5-th column of the above M1. The black line crosses the regions with sign-pattern (−, 0, +)⊤ and (+, 0, −)⊤. At least these two regions have been crossed by two planes. Due to the fact that we know that J11 < 0, for the next step we need to count the number of the remaining hyperplanes that cross the region with the sign-pattern (−, 0, +)⊤. In Supplementary Fig. 5b we find that four of the remaining hyperplanes cross this intersection line, that is, the normalized ϕ satisfies ϕ =1. It means that the sign-pattern of this intersection line is the inference of sign(J1) because it meets the requirements of sign-satisfaction for all the sample pair.
2.3.4. The heuristic algorithm combing sign-satisfaction and intersection of hyperplanes
Combining the sign-satisfaction graph with the sampling procedure described above, we propose a heuristic algorithm to infer the sign-pattern of Ji.
Our heuristic algorithm has two inputs: the steady-state dataset for the i-th taxon and a user-defined parameter Ψ determining how many intersection lines of hyperplanes will be constructed. The algorithm has of four steps, as described in Supplementary Fig. 6. Applying this procedure for i = 1,…,N, we can get the sign-pattern of the whole Jacobian matrix.
In summary, the algorithm works as follows. After generating an intersection line, we get the three orthants corresponding to this intersection line. Then we count how many hyperplanes cross the orthants determined by this intersection line using the sign-satisfaction graph, and this count can be normalized as ϕ ∈ [0,1] indicating the confidence of this potential solution to be a solution of final inference. Finally, we select the intersection line with the maximal ϕ among the generated Ψ intersection lines as the final inferred sign-pattern .
Note that if the algorithm is stuck in generating an intersection line for some subset of (N − 1) hyperplanes, the heuristic algorithm will fail. Numerical experiments suggest this situation happens only when the data is not informative enough or the number of samples is smaller than the threshold Ω*. In Fig. 3 of the main text, we presents the results of the minimal number of samples Ω* required for a community with size N.
2.4. Limitations of the inference when using relative abundance data
High-throughput amplicon sequencing of 16S RNA has become a well-established approach for profiling microbial communities. The result of this procedure are measurements of the relative abundance of each taxa in the microbial community, meaning that these quantities have been normalized to sum to one (or some other arbitrary constant). This implies that an increase of the relative abundance of one taxon must be accompanied by a decrease in the relative abundance of other taxa. This severely limits the application of system identification methods based on temporal data, as discussed with details in [8, 9].
The use of steady-state samples containing relative abundance also leads to inference errors. Consider, for example, that there exists three relative abundance profiles containing taxon i, say . Since they are relative abundances, the sum of each of these samples must equal 1. This also implies that . Consequently, the vector satisfies
In other words, this vector 1 is always in orthogonal to all sample differences and the intersection line of the (N − 1) hyperplanes generated by relative abundance is always 1. Therefore, the heuristic algorithm fails in correctly inferring the sign-pattern of Jacobian matrix using relative abundances, because it always predicts that one possible sign-pattern is sign(1) = (+,…, +).
3. INFERRING THE TOPOLOGY OF ECOLOGICAL NETWORKS
In essence, inferring the zero-pattern is similar to inferring the sign-pattern. Indeed, it is only necessary to recognize any non-zero entry of the inferred sign-pattern as a non-zero entry in the inferred zero-pattern. Notice how the zero-pattern corresponds to hyperplanes exactly aligned to the orthants of . Therefore, any measurement noise will make the difference of sample pairs deviate from the axis, easily leading to inference errors (see Supplementary Fig. 1d,f). To alleviate this problem, we introduce a user-defined cutoff value to judge the zero-pattern of Jij based on the angle between the axis and the intersection line (Supplementary Fig. 1).
For the brute-force method, first we set an element in the difference of sample pair xI − xK to 0 if the magnitude of that element is less than the user-defined cutoff. Second, we construct the hyperplanes respectively orthogonal to these modified difference of sample pairs. Third, we count how many hyperplanes cross each orthant in the . Finally, we select the region crossed by the maximal hyperplanes as the inferred zero-pattern. Recall that the brute-force method is limited to infer the microbial community with N ≤ 10.
For larger microbial communities, we also developed a heuristic algorithm that is very similar to our heuristic algorithm for inferring the sign-pattern. In that algorithm, notice how the deviation of an intersection line from an axis is directly given by its directional vector. Indeed, this vector contains the cosine of the angles between the axis and the intersection line. This algorithm works as follows:
Construct the matrix of the difference of all the sample pairs. Consider the set of all vectors . Let be a matrix constructed by stacking all the vectors, where is the number of samples containing taxon i. By construction, each column of Mi is the normal vector of a hyperplane orthogonal to the difference of the corresponding sample pair.
Randomly sample (N − 1) hyperplanes. Choose randomly N − 1 columns from Mi.
Find the intersection of the (N − 1) sampled hyperplanes to obtain an intersection line. This can be done by finding the kernel of the matrix obtained by stacking the chosen columns. Note that the randomly sampled (N − 1) hyperplanes not always intersect in a line, because some hyperplanes might be parallel. However, this situation is non-generic in . Thus, if the randomly sampled hyperplanes do no intersect as a line, we return to step 2 and Choose a new subset of columns.
Set the elements in the directional vector of this intersection line as zero if their absolute values are less than the cutoff value. Then we get a new directional vector. Note that we scale the 2-norm of this directional vector to 1. The absolute value of i-th element in the directional vector represents the cosine of the angle between the intersection line and xi-axis. If the value is large enough, it means the intersection line almost locates at the xi-axis. Therefore, if the absolute value of directional vector is smaller than the cutoff, we set this entry to 0.
Count how many hyperplanes cross the region of the new directional vector using the sign-satisfaction graph. The sign-pattern of new directional vector represent the orthants in crossed by all those (N − 1) hyperplanes. For the rest hyperplanes of Mi (i.e., the rest of the columns in Mi), let be the number of those hyperplanes that cross the orthants. We normalize using , so that ϕ ∈ [0,1]. Notice that ϕ = 1 means that the sign-pattern of the new directional vector meets the requirements of sign-satisfaction for all the sample pairs. Therefore, the magnitude of the computed ϕ can be seen as the confidence of this potential solution to be a solution of the sign-satisfaction problem.
Go back to Step (2) until Ψ ≥ 1 intersection lines have been computed.
We validated this method using steady-state data generated from four different population dynamics. Except the Generalized Lotka-Volterra (GLV), the other three population dynamics models have non-linear functional responses: Holling Type II (H), DeAngelis-Beddington (DB) and Crowley-Martin (CM). Supplementary Fig. 7 shows the inferred network topology on four different population dynamics models. We found that in case the noise level is η = 0.1, the accuracy of inference can be around 0.8, if the cutoff is between 0.1 and 0.2. However, in the noiseless case, increasing the cutoff can decrease the accuracy, because larger cutoff induces more false positives of interactions.
4. INFERRING THE ECOLOGICAL INTERACTION TYPES
Using the brute-force method to infer the interaction types is deterministic because we search all the combinations of {+, 0, −}N, and accuracy increases with the increment of sample size. However, due to the time complexity, application of the brute-force method is limited to small microbial communities, e.g., N ≤ 10. This motivated us to develop the heuristic method of Supplementary Note 2 that is suitable for larger microbial communities.
To validate the effectiveness of our heuristic algorithm, we tested it using simulated steady-state data generated by models of the form Eq. (S1). In particular, we considered a model with pair-wise interactions of the form where is the intrinsic growth rate of the i-th taxon, is a constant matrix and the function is the so-called functional response [1, 2, 3, 4, 5, 6]. Recall that these functional responses model the intake rate of a consumer as a function of food density, and thus different functional responses correspond to different mechanisms of interaction between taxa.
We used Eq. (S9) to generate synthetic steady-state datasets for 4 different functional responses with different complexity. The first was the linear functional response for which Eq. (S9) actually reduces to the classical Generalized Lotka-Volterra (GLV) model. In this case, the accuracy of the heuristic algorithm on inferring the sign-pattern sign(J) = sign(A) is 100% if there are enough steady-state samples, see Fig. 3a in the main text. Indeed, this is a consequence of the following proposition:
In the noiseless case, if the functional response is linear, the directional vector of intersection line of any (N − 1) hyperplanes orthogonal to is the same and parallel to Ji.
Proof. Due to the fact that the functional response is linear, the Jacobian matrix become simple and constant for different samples, that is, J = A. Therefore, Eq. (S3) is equal to where denotes the difference of all sample pairs. As we know, ai ≠ 0, representing the interaction vector in the A matrix, is unique to Mi. Thus the non-trivial solution of ai in the above equation array must meet the requirement
That is to say, if we randomly select (N − 1) columns in Mi as , then
Actually, the randomly selected corresponds to (N − 1) hyperplanes respectively orthogonal to each columns of in the geometric perspective. The directional vector of intersection line of these (N − 1) hyperplanes can be calculated by null , which is parallel to ai.
The remaining three functional response were Holling Type II (H), DeAngelis-Beddington (DB) and Crowley-Martin (CM), given by the following equations
Here c1, c2, c3 are constants. Note that these nonlinear functiunal responses lead to more complicated population dynamics. For the results presented in Fig. 3 of the main text, we used c1 = 1, c2 = c3 = 0.1. Thuse results show that the heuristic algorithm accurately infers the sign-pattern of Jacobian matrix for these three functional responses and its accuracy is above 95%.
5. INFERRING INTERACTION STRENGTHS WITH GLV DYNAMICS
A particular class of systems in (S1) is when the Jacobian Ji is constant, implying that for some constant vector and scalar ri. In such case, the system reduces to the Generalized Lotka-Volterra (GLV) model where is the i-th row of the so-called interaction matrix , and is the intrinsic growth rate of taxon i. As discussed in the main text, the GLV models also allows defining the interaction strength of taxon j on taxon i as aij.
5.1. A condition for detecting GLV dynamics
Our first observation is that the steady-state samples can be used to decide if they could be produced by a GLV model:
A necessary condition for the dynamics of the i-th species to be GLV is that all samples align into a hyperplane.
Proof. If for all the samples xI align into a hyperplane, then fi(x) should be a hyperplane whose general equation is .
As discussed in the main text, with real data containing measurement noises and other errors, the samples will not align exactly into a hyperplane. In such case, the coefficient of determination (denoted by R2) of a hyperplane fitted to the samples containing taxon i can be used to judge if its dynamics can be adequately described by the GLV model. For a given dataset , if the average of R2 of the hyperplanes fitted to the samples of the i-th taxon is > 0.9, then we consider that it is possible to infer the inter-taxa interaction strengths and intrinsic growth rates using the GLV model for this taxon. Otherwise, we recommend to infer only the interaction types. The pipeline for detecting GLV dynamics is described as Supplementary Fig. 8.
5.2. Inference of interaction strengths and intrinsic growth rates
Under the GLV model, Eq. (S3) reduces to
If we denote by Pi the (N − 1) dimensional hyperplane spanned by all the steady-state samples sharing the i-th taxon , Eq. (S11) implies that the ai belongs to the one-dimensional space orthogonal to Pi. Thus the normal vector of the fitted hyperplane according to is parallel to ai. To infer the precise value of interaction strengths, additional prior information, at least one non-zero element in ai, is needed. Otherwise, we can only infer the relative strength of the interactions between taxa.
5.3. Applying the Knockoff filter to control the false discovery rate
Eq. (S11) shows that ai can be inferred by fitting a hyperplane based on all the steady-state samples sharing the i-th taxon , provided that we know at least one non-zero element in ai, say aii (or an estimate of it). Consider that the ecological network to be inferred is sparse. Then, a natural method to find a sparse solution is by using the so-called Lasso regression: where λ is the Lasso (regularization) parameter. Here y is the i-th column of , and is the matrix obtained from by deleting the i-th column and adding 1 in the end. xi is the i-th column of . This structure happens because for the GLV we have ri + ai1x1 +…+ aiixi + ai,i+1xi+1 +…+ aiNxN = 0 and we assumed for the numerical results that aii = −1. Once a solution β to the above Lasso problem is found, the estimation for ai is given by where β(i0: if) is the vector obtained by concatenating the elements i0 to in of the vector β. Recall that the parameter λ in the Lasso is crucial for accuracy. A classical method to optimally choose this parameter is using cross validation.
However, even after using cross validation, the Lasso tends to induce a high false discovery rate (FDR), i.e., many zero interactions are inferred as non-zeros ones. Formally, the FDR of a inference procedure y = Xβ + z, returning the inferred parameters , is defined as
Here a ⋁ b = max{a, b}.
Recently, the so-called Knockoff filter has been proposed as an enhancement to the Lasso algorithm to maintain the FDR below a certain user-defined level q > 0, regardless of the value of the coefficients β (see [10]). This method works by constructing the so-called “knockoff variables” that mimic the correlation structure found in the real data. The knockoff copy of each variable act as a “control group”, allowing to assign a “trust” to each inferred variable. It has been shown this strategy successfully controls the FDR. In our work we used the Matlab package of the Knockoff filter as provided in https://web.stanford.edu/~candes/Knockoffs/package_matlab.html. The validation of the network inference with GLV dynamics is shown in Fig. 4 of the main text.
5.4. Blinded inference of interaction strengths by assuming GLV dynamics
Here we show that, if the steady-state samples were collected from a microbial community without GLV dynamics, the inference of interaction strengths by assuming GLV dynamics systematically leads to inference errors.
To illustrate this point, we first generated steady-state samples using Holling Type-II functional response. Then, we applied the GLV-based inference method to the steady-state samples in order to infer the interaction strengths. Supplementary Fig. 9 shows that the accuracy (the percentage of correct sign of the inferred interaction strengths compared with the sign of ground truth) of inferred results is very low, even in the absence of noise. This is consistent with the small value of R2 of fitted hyperplanes, which describe the deviations of samples to those fitted hyperplanes. This suggest that inferring the interactions strengths of a real microbial community without first testing if its dynamics can be described by the GLV model can produce significative errors.
6. REAL DATASETS
6.1. A synthetic microbial community of 8 soil bacteria
In [11] a set of eight heterotrophic soil-dwelling bacterial species were studied for predicting species persistence in different assembled microbial microcosms. The steady-state dataset consists of a total of 101 different species combinations: 8 solos, 28 duos, 56 trios, 8 septets and 1 octet (Supplementary Fig. 10a). Each species combination of cultivation was carried out in duplicate and started from different configurations of initial abundance. We averaged the steady states from different initial conditions.
First we find that R2 of each fitted hyperplane is less than 0.9 (Supplementary Fig. 10b), which indicates that this microbial community could not be properly described by the GLV model. Hence we focus on the inference of interactions types between any two species. To be fair, without considering the 8 solos and 28 duos, we analyze the rest steady-state samples. We use both the brute-force algorithm (see Fig. 5 of main text) and the heuristic algorithm (Supplementary Fig. 10c,d) to infer the ecological interaction types. In Supplementary Fig. 10c, blue (or red) means inhibition (or promotion) effect of species j on species i, respectively. We found that 11 signs were falsely inferred, 5 signs were undetermined by the analyzed steady-state samples. The inferred results are very similar with the brute-force method shown in Fig. 5b of main text. Furthermore, Supplementary Fig. 10d shows that once Ψ is larger than a certain value, the accuracy in the inference does not increase any more.
6.2. A synthetic community of maize roots with 7 bacterial species
(Fig. 6b in the main text). There are in total 7 bacterial species (Ecl, Sma, Cpu, Opi, Ppu, Hfr and Cin) in this community [12]. The available steady-state data consists of 7 sextets (i.e., data from seven experiments in which six different species grow together) and 1 septet (i.e., data from one experiment in which the seven species grow together). This leads to a total of 8 steady-state samples, see Supplementary Fig. 11a.
First, based on our theoretical result showing that in the generalized Lotka-Volterra (GLV) model the steady states that share common species will align into a hyperplane, we concluded that this bacterial community does not follow the GLV dynamics (see Supplementary Fig. 11b). Thus, we have to focus on inferring the interaction types, rather than interaction strengths.
Second, only using the 7 sextets we inferred the sign-pattern of the Jacobian matrix (Fig. 6a in the main text). Based on the inferred sign of Jij, we can predict how the abundance of species i will change, when we add species j to the community (see results in the Main Text).
6.3. A synthetic microbial community of two cross-feeding partners
In this community [13], two non-mating strains of the budding yeast, Saccharomyces cerevisiae, were engineered to be deficient in the biosynthesis of one of two essential amino acid tryptophan (Trp) or leucine (Leu), and to overproduce the amino acid required by their partner. It has been demonstrated that these two strains form a community with cross-feeding mutualism, where each strain provides the amino acid needed by its partner. In [13], the authors inoculated monocultures and co-cultures at a range of concentrations of supplemented amino acids in a well-mixed liquid batch. Supplementary Fig. 12a-c shows the abundance of the co-cultures and monocultures for the Trp and Leu strains at low, medium and high levels of supplemented amino acids. After 7 days cultivation, the abundance of each species approaches its steady state. Note that for each scenario, the experiments inoculate a constant amount of resources at the beginning. Here the type of interaction is defined by comparing the abundance of co-cultures with monocultures at the end of cultivation. As the supply of amino acids increases from low, to medium to high concentrations, the interaction between this pair of strains shifts from obligatory mutualism (Supplementary Fig. 12a), to facultative mutualism (Supplementary Fig. 12b), and to parasitism (Supplementary Fig. 12c), respectively.
We applied our inference method to each scenario. Supplementary Fig. 12d-f shows the diagrams of our inference results that are consistent with the empirical observations. For example, in Supplementary Fig. 12e,f, the cyan line orthogonal to the red line is very close to the Leu axis, which indicate the effect of Trp on Leu is very weak. Especially in Supplementary Fig. 12f, this promotion effect can be ignored.
6.4. A synthetic community of 14 auxotrophic Escherichia coli strains
Starting from a prototrophic E. coli derivative MG1655, the authors of [14] generated 14 strains, each containing a gene knockout that lead to an auxotrophic phenotype unable to produce 1 of 14 essential amino acids. By convention, the authors labeled each auxotrophic strain by the amino acid it lacks. For example, the methionine auxotroph ΔmetA auxotroph is strain M. It was confirmed that the 14 auxotrop (C, F, G, H, I, K, L, M, P, R, S, T, W, Y) show no growth in M9-glucose minimal media after 4 days. Indeed, they grow only when supplemented with the essential amino acid they were not able to produce. This dataset consists of co-cultures of all 91 possible strain pairs from the 14 characterized auxotrophic strains. For each pairwise co-culture, we are able to calculate the total fold growth, i.e., the yield of the community calculated by (total final cell density)/(total initial cell density), as well as the fold growth of each strain. Since these auxotrophic strains cannot grow by themselves, if strain i is able to grow as a co-culture when paired with strain j, and strain i’s fold growth is Fij > 1, this implies that strain j promotes the growth of strain i, i.e., Jij > 0. By contrast, if Fij < 1, we cannot conclusively say that Jij < 0 because we lack the monoculture data. Therefore, the fold-growth metric can only be used to detect a promotion effect between two strains.
First, we found that R2 of all fitted hyperplanes are smaller than 0.9, implying that the population dynamics of this microbial community cannot be properly described by the GLV model (Supplementary Fig. 13a). Second, we used the heuristic algorithm to infer the interaction types (Supplementary Fig. 13b). Note that the complexity of the inference approaches 314 ~ 4 x 106 if we use the brute-force algorithm. We found that the types of 14 pairwise interactions cannot be determined with the given dataset (marked in gray in Supplementary Fig. 13b). Third, we showed the fold growth matrix F = (Fij) from experimental observations (Supplementary Fig. 13c), with Fij the fold growth of strain i (row) in the co-culture paired with strain j (column). Here we set Fij ≥ 20 as an indication of promotion effect of strain j on strain i. There are in total 71 promotion interactions with such a large confidence (shown in red, Supplementary Fig. 13c). We will use them as the ground truth to check our inference results on promotion effects (i.e., positive signs, shown in red in Supplementary Fig. 13b). We found we inferred 13 wrong positive signs (marked as ‘×’ in Supplementary Fig. 13c), and missed 5 positive signs (marked as ‘?’ in Supplementary Fig. 13c). Therefore, our inference of positive signs has an accuracy of 74.65% (53/71), if we set the fold growth threshold 20 as the indication of promotion effect. We also observed that the accuracy on the inference generally increased by increasing this threshold (Supplementary Fig. 13d).
7. RELATIONSHIP TO EXISTING NOTIONS OF INTER-TAXA INTERACTIONS
In Assumption 2, we considered that the Jacobian of (S1) determines the interaction types between microbial taxa. This assumption was then used to build our network reconstruction method. Here we discuss how this consideration compares to other existing definitions and notions of “interactions” available in the ecological literature.
In general ecological systems, understanding the interactions between taxa and their strengths is key for developing predictive models and conservation strategies. This has motivated the introduction of several empirical indices for inter-taxa interactions, specially for consumer-prey ecosystems [15, 16]. Let x1 and x2 denote the abundances of prey and consumer, respectively. Consider two samples for this ecosystem consisting of an experiment with the prey in isolation —that is, with the consumer or predator deleted— and other with both prey and consumer present . Here we discuss the four empirical indices as used in [16]:
Raw difference:
Paine’s index:
Community importance: , where
Dynamic index: , where t is time.
All the above indices have identical signs, solely determined by . As shown in Example 3, such sign coincides with one of the possible sign-patterns obtained by applying our reconstruction method for N = 2 taxa. More precisely, the above indices coincides with our reconstruction method provided we assume this as prior information. Such prior information can be interpreted as adopting a “convention” for the sign of self-interactions (i.e., a kind of “relative sign-pattern”).
Compared to the analysis in [16], our reconstruction method provides more general conditions under which the above indices provide the correct sign of the interactions according to a mathematical model.
Our reconstruction method also generalizes the application of the above indices to ecosystems with an arbitrary number of taxa, and beyond the consumer-prey interactions.
Our reconstruction method provides conditions under which the available steady-state data is informative enough to infer the correct sign of a desired microbial interaction.
According to our framework, note there are two different interactions that is possible to infer: x1 → x2 and x2 → x1. The above indices and discussions are concerning the interaction x2 → x1 —that is, the effect of the consumer on the prey. In order to infer the sign of the interaction x1 → x2, we need to evaluate . In the case of consumer-prey ecosystem with N = 2 taxa, it might be impossible to measure a non-zero x{2}, since it corresponds to a steady-state abundance of consumers in the absence of prey. In such case, the set contains only one sample x{1,2}, and thus the given data is not informative enough to infer this interaction. This argument could explain cases when for N taxa it is impossible to infer some interaction due to the absence of the needed sample, simply because in the absence of some taxa other become extinct.
Acknowledgements
This work is supported in part by the John Templeton Foundation (Award number 51977). We thank Drs. Gabe Billings and Brigid Davis for insightful comments on the manuscript. We thank Drs. Joseph Nathaniel Paulson, Michael T. Mee, Harris H. Wang, Francesco Carrara and Carsten F. Dormann for kindly providing their experimental datasets. We thank Dr. Liang Tian for discussions.
Footnotes
↵1 We say a steady state is feasible if it belongs to the orthant , that is, if no taxon has negative abundance.