Help understanding number of samples' impact on Maaslin results

Hello,

This is probably a very naive question, but I have a hard time explaining this difference in results, and thus deciding what is the best strategy to adopt.

I am analyzing mice gut microbiome data (16S and WGS) with Maaslin3. I have a Treatment variable for which I have a Control group and multiple doses tested. I first wanted to make sure the compound had an impact on the gut microbiome. To do that, my naive strategy was to subset my data to keep only the control samples and the samples who had the highest dose. The rationale behind this subsetting was to reduce the number of test by reducing the number of features, as we keep only taxa present in at least x samples of our subsets (typically between 2 and 5 samples), to increase statistical power. This type of analysis could not select any taxa associated with the treatment, which lead us to explore other DA methods like ANCOM-BC or older methods like Lefse, because we could see, visually, a lot of differences, but our mice groups being very small, we figured that maybe Maaslin was too stringent.

Because we had different doses, I later decided to test whether there were a dose-effect, despite the lack of significant results of our first analysis. I conducted 2 analyses : 1- each dose was compared to the Control (Taxa ~ Treatment + (1|Cage)) and 2- each dose was compared to the prior dose (Taxa ~ ordered(Treatment) + (1|Cage)). To my surprise, and even though I had twice as many taxa tested in this analysis, several taxa were now selected as significantly associated to the highest dose, for both analyses. I have a hard time understanding what leads to this difference, and how, even though we significantly increased the number of tests, we have better statistical power at the end.

Thank you for your help

Have a nice day

Hi,

A few thoughts:

  1. When you say you can see differences visually but the DA methods aren’t detecting anything, what are you visualizing that shows a difference? Typically, any effect you can see in a scatterplot or box plot shows up extremely strongly in MaAsLin 3. Much more often, the visuals-vs-stats comparison is confusing because MaAsLin 3 detects differences but the plots don’t show them to be that obvious. This makes me think there might be something off in either the visualization (e.g. per-mouse grouping not accounted for) or what you’re putting into the DA methods if there’s a strong visual effect you don’t see in the statistical results.
  2. What is the experimental design? How many mice per group, how many doses, how many samples per mice, was it one dose per mouse? The numbers here can have a pretty substantial impact on which analyses are feasible and which you probably won’t have power for.
  3. In our evaluations while developing MaAsLin 3, using ordered() or group() with a random effect tended to increase false positives quite a bit more than any other combination of parameters because highly constrained fits get more likely as the model gets more complex. It’s probably worth nailing down (1) and (2) first before hand-wringing about whether the significant associations with the ordered + random effect model are just false positives.

Will

Hello,

Thanks for your reply,

The way I visualize the taxa abundance is by selecting the 20 most abundant taxa of my dataset (I computed the mean abundance of each taxa and selected the 20 most important across my subset containing only the control and highest dose group), and checking with stacked barplots how these 20 taxa looked between the control group and the highest dose group, showing one bar per sample, so I can see the individual differences. There is, of course, a lot of within group differences, but, for example, we can clearly see that the samples who received the highest dose tend to have more Klebsiella, while the control group tends to have more Escherichia, so there is kind of a shift happening there. But initially could not detect Klebsiella or Escherichia as being differentially abundant in the highest dose (we can when I supply all the doses afterwards). I can’t really show plots in public unfortunately.

I talked about mice because I have several datasets with this issue, but the one for which we see the most differences is actually an in vitro dataset of human stool samples cultivated in a gut model. For the experimental design we had one control group, a low dose group, medium dose and high dose group. Each group had 8 donors (so 8 samples per group). And yes, each samples received only one dose.
My maaslin code looked like this:
for the model that did not show any significant association:
maaslin3(
input_data = count_table_medium_highdose_t,
input_metadata = metadata_medium_highdose_log,
fixed_effect = c(“Treatment”, “Nb_reads”),
random_effects = c(“Donor”),
small_random_effects = T,
reference = c(“Treatment,Medium”),
output = output_path,
normalization = ‘TSS’,
transform = ‘LOG’,
augment = TRUE,
standardize = TRUE,
)

for the models that showed significant associations:

maaslin3(
input_data = t(count_table_maaslin),
input_metadata = metadata_maaslin,
formula = “~ Treatment + Nb_reads + (1|Donor)”,
reference = c(“Treatment,Medium”),
small_random_effects = T,
output = output_path,
normalization = ‘TSS’,
transform = ‘LOG’,
augment = TRUE,
standardize = TRUE,
)

or

maaslin3(
input_data = t(count_table_maaslin),
input_metadata = metadata_maaslin,
formula = “~ ordered(Treatment) + Nb_reads + (1|Donor)”,
small_random_effects = T,
output = output_path,
normalization = ‘TSS’,
transform = ‘LOG’,
augment = TRUE,
standardize = TRUE,
)

I realised I passed the small_random_effects = T to all my models, which was maybe unnecessary, but I dont know if that alone could explain the differences in results?

Have a great day

My maaslin code looked like this:
for the model that did not show any significant association:
maaslin3(
input_data = count_table_medium_highdose_t,
input_metadata = metadata_medium_highdose_log,
fixed_effect = c(“Treatment”, “Nb_reads”),
random_effects = c(“Donor”),
small_random_effects = T,
reference = c(“Treatment,Medium”),
output = output_path,
normalization = ‘TSS’,
transform = ‘LOG’,
augment = TRUE,
standardize = TRUE,
)

for the models that showed significant associations:

maaslin3(
input_data = t(count_table_maaslin),
input_metadata = metadata_maaslin,
formula = “~ Treatment + Nb_reads + (1|Donor)”,
reference = c(“Treatment,Medium”),
small_random_effects = T,
output = output_path,
normalization = ‘TSS’,
transform = ‘LOG’,
augment = TRUE,
standardize = TRUE,
)

Are you saying the first of these produced no significant associations but the second did? If I’m reading it correctly, those should be the same model, so I would expect they’d produce the same results.

I realised I passed the small_random_effects = T to all my models, which was maybe unnecessary, but I dont know if that alone could explain the differences in results?

If I understand this correctly that each mouse is sampled once and your random effect is by donor with 8 samples per donor, 8 should be enough (>=5) that you don’t need to use the small random effects, which will probably help a bit with power.

Hi,
Yes, the first Maalsin command did not show any significant association, but the second did. The only difference is that, for the second command, the data passed to Maaslin contains samples for every doses, while the first model was only ran on a subset of the data, containing only the control and highest-dose samples. There are twice as many taxa in the data passed to the second command. I checked and the taxa for which it founds a significant association were not filtered out in the subsetted data passed to the first command. I too would have expected similar results between the outputs of the two commands, for the highest dose group.

I will test tomorrow re-running these commands without the small_random_effects argument when I am back in the lab, but I would not expect this to change the results that much, or could it ?
Thanks for the help
Best regards

Ah, got it. If the sample size is quite a bit higher in the second one, it would make sense that it has more power to detect associations. Setting small_random_effects to false won’t change the abundance associations, but it can make the prevalence associations quite a bit stronger especially if you have 8 samples per group.

Out of curiosity, if you don’t do any taxa filtering at all, how different do the results come out? We’ve found that MaAsLin 3 has similar power regardless of whether you filter the taxa beforehand, so I wonder if the filtering is making something strange. For example, if you renormalize the samples after dropping taxa that are only in a few samples each but when present account for much of the abundance, the renormalized abundances could be quite different from what you started with. Our recommendation in the MaAsLin 3 docs is to not do any taxa filtering beforehand.

I get that increasing the number of samples increases statistical power (I have twice as many samples when I supply all the doses to maaslin), but it also significantly increases the number of features tested. Because of that, I would expect that finding associations when comparing my highest-dose group to the control may work better when working with subsets, instead of the whole dataset. That is where I think I am lacking something in my understanding of this method, would supplying more samples (even if they are samples from other dose-group) always be a better choice to find association to the highest-dose, compared to subsetting to only keep the samples from the groups of interest ? And how exactly are the samples from the lower and medium dose groups helping to find the association for the high-dose group ? Sorry this is coming down to some very basics aspect of Maaslin I realise, but I might be missing something here.
I will test feeding my data without any taxa filtering tomorrow and I’ll tell you how it goes. It’s true we tend to increase taxa filtering threshold when we lack statistical power (few samples per groups, very noisy datasets), and with the normalization step within Maaslin, I see how it could create some artefacts. But, if we were to continue our taxa-filtering step before Maaslin (sometime we filter out taxa that are absent in more than 50% of the samples of a chosen group), would it be sensible to disable the normalization arguments, while making sure that we supply a table of relative abundances, that were computed before the taxa filtering step?
Thanks a lot

Increasing the number of associations tested is only a problem if you’re doing false positive correction with something like Bonferroni. By contrast, if you’re using Benjamini Hochberg (the default in MaAsLin 3) to control the false discovery rate, it doesn’t matter whether you double the number of associations, as long as their significance values have the same distribution. If you dump in a bunch of null associations, the corrected significance values will go down, but the other dose groups you’re testing probably also have some effect, so you’re probably preserving the same distribution of p-values going into the correction.

Having the extra samples can actually help because it allows the linear model to better estimate the shared variance parameter (since in linear models the variance is assumed to be equal across all the groups).

If you’re going to continue with the taxa filtering step, I’d compute the relative abundances with all the samples unfiltered, then do the filtering, then put the remaining profiles into MaAsLin 3 with normalization set to NONE so that it doesn’t renormalize what’s left.

Hi,
sorry for the delayed response, I was trying different stuff on several datasets after our discussion.
Removing the small_random_effects improved quite substantially with statistical power, I got more significant associations, for both prevalence and abundance models, which is a bit unexpected? In my new results, I always get one or two prevalence associations with a very strong effect (absolute coefficient of about 10-20), which seems weird to me, as I rarely get such strong signal in my analyses, for this type of model, do you think it is odd, or not necessarily?
I tried feeding the count table without any taxa filtering to Maaslin and it yielded similar results overall, sometimes some associations had a slightly lower q-value. But I tested it on datasets for which I did not do a lot of filtering in my previous Maaslin analyses. In any case, I will be more mindful of taxa filtering and the internal normalization step happening in Maaslin in the future. I realise, as time goes, and as I use Maaslin for a lot of different datasets, I tend to forget good practice and think less of the consequences of some pre-Maaslin transformations steps.
About the impact of adding more samples on statistical power, we initially assumed, from previous results, that the intermediate doses had no effect, and that adding them would probably lead to a decrease in statistical power, but Maaslin analysis proved that to be false, and I stayed on my first assumption despite that.
Thanks a lot for the discussion, even though in the end, my issue was mostly due to the small_random_effect being passed to Maaslin when it shouldn’t have, it helped me to take a step back on the way I ran some analyses, which will be helpful in the future, so I greatly appreciate your insights.
Best regards

Glad that worked out! One note of caution when not using small_random_effects is that you can occasionally get p/q-values that are extremely small (e.g. <10^-13) due to highly constrained model fits. The abundance associations won’t have this problem, but for any prevalence associations that are extremely significant, I’d check that you have a reasonable balance of presence/absence across the groups and that the results aren’t entirely driven by a small number of data points.

Hi,
Sorry to come back to this discussion several weeks later, but I would have a couple more questions.
Removing the small_random_effects argument does help gaining statistical power for the prevalence model, even though we still have a bunch of abundance association that are, when we check the abundance itself, clearly a prevalence thing. But it is also true that it introduces the risk of having some prevalence association appearing very strong/significant even though, when we look at the data, they are at best small variations, or even completely irrelevant. In fact, for the dataset we were discussing, we have this issue for every maaslin command I ran (depending on the subsets I was working on), with always at least one association having very strong effect (values ranging from 10 to 20) and veeeery low p-values, that, when looking closely at the abundance/prevalence table, are actually either mild effects (way less important than other community shifts appearing less relevant in Maaslin results) or not relevant at all. Do you have any idea of why this might happen, or how I could prevent that with the current model I am working with?

Furthermore, for this dataset, we are also interested on the effect of probiotics on both the community at basal level and depending on Treatment. So basically for this part of the nalaysis we have a Treatment variable (Control, Low, Medium, High) and a probiotics variable (Absent or Present). My model was

~ Treatment * probiotics + Nb_reads + (1|Donor)", 
reference = c("probiotics,Absent;Treatment,Control")

I was reading on how to properly interpret the results we get from these kind of models (mainly the interaction terms) (like here). From my understanding, If I want to interpet an interaction such as TreatmentMedium:probioticsProb1, since the effect is not the direct impact for both the treatment at medium level and the probiotics presence, I need to compare the effect of this interaction term to the probioticsProb1’ effect size, as it show the effect of the probiotics in the control group, right? I guess my main question is how to interpret the direction of the coef of the interaction terms.
Does that mean that, if I have a negative coefficient for an abundance association for TreatmentMedium:probioticsProb1, I can consider that, at this dose, the treatment has an opposite effect on this taxa’s abundance, compared to the condition TreatmentMedium without probiotics? Or does it mean that the abundance is simply decreasing compared to TreatmentMedium without probiotics? Because I was assuming it was the first explanation, I wanted, for every taxa with a significant association to an interaction term, to fetch the coefficient of its association to the Probiotic (So effect of probiotics on this taxa, on the Control group) to see what its direction is, but then I will be working with a lot of non-significant associations. I guess I am wondering how much we can infer from these interaction terms, for my kind of model, and what we can’t?

Thanks a lot
Have a great day