A conversation with a colleague
A few weeks ago, I was chatting with a colleague about one of his ecological field experiments. His group is testing the impact of a biocontrol insect on the growth of an invasive tree species. They released the insect at four sites on the Big Island about a decade ago and have been tracking changes in tree diameters at those locations annually.
But, I was surprised to learn that they are not tracking tree growth at sites that have not received the biocontrol. I asked what they were comparing the growth rates against, and he explained that for several years before the insects were released, they had measured tree diameters at the same four sites. So, rather than comparing tree growth between control and treatment sites, they were comparing growth at the same sites before and after the treatment.
But what if something else besides the insects was contributing to differences in tree growth rates over time? The current study design would not be able to distinguish those separate factors. I brought up this point with him as well. He admitted that having such control sites would be ideal. But he also explained that in ecological experiments, it’s not always feasible–due to constraints like limited funding and spatial variability–to implement the ideal experimental design. Sometimes, compromises have to be made.
Experimental study designs in ecology
Although I was not totally ignorant to the challenges of carrying out ecological experiments, most of my experience with scientific experiments has taken place in the lab or greenhouse: settings that are much smaller and uniform in scale, that are much easier to control, and where more sophisticated experimental designs are feasible to implement.
This got me interested in learning more about experimental design in ecology. Using Claude, I found a 2019 paper by Christie and colleagues titled, “Simple study designs in ecology produce inaccurate estimates of biodiversity responses“. In this study, the authors tested the effects of different study designs on the accuracy of estimating changes in population density as a result of some impact (e.g., experimental treatment, disturbance event, etc.).
They tested the following designs:
- After: measurements taken only on the impacted areas and only after the impact
- Before-After (BA): measurements taken only on the impacted areas, both before and after the impact
- Control-Impact (CI): measurements taken on both impacted and control (i.e., unimpacted) areas, but only after the impact
- Before-After Control-Impact (BACI): measurements taken on both impacted and control areas, both before and after impact
- Randomized Controlled Trial (RCT): like CI, but experimental units are randomly assigned to either control or impact groups.
The table below from the Christie et al. paper illustrates these different designs.
Figure 1
“Comparison of the key features of study designs. Graphs show how designs sample from impact (green points) and control (blue points) sites over time, before and after an impact (white vs. grey areas, respectively). Solid horizontal lines show the average density of sites measured to calculate each design’s effect size estimate. Dashed horizontal lines for CI and RCTs represent the pre-impact differences between the mean densities of control and impact sites, which can cause bias – note less difference for RCTs (with high sample size) versus CI. Many design variants exist – e.g. MBACI for BACI with multiple sites, R for Reference in BARI (Webb et al., 2012)”
What would these study designs look like in my recent home garden studies?
If you’ve been keeping up with this blog over the past summer, you’ve probably seen at least one article about my garden. I did a series of four articles looking at plant biodiversity and related topics. Now, I’m going to walk through the study designs listed in the last section, one-by-one, and explain how each design would look if applied to my garden questions. I find it helpful when learning a new concept to apply it to something I’m already familiar with.
Before I dive in, it’s worth noting that my garden inventory example differs from the examples in Christie et al. in two key ways. Whereas I took only single measurements before and after the impact (i.e., gardening) and measured the entire population, the examples in the paper assume multiple measurements of samples (i.e., subsets of populations).
- After: In this scenario, I would have only taken an inventory of my garden after I started gardening. I would still be able to calculate diversity indices, such as the Hill-Shannon index. But I would not be able to say how much the diversity changed as a result of my gardening, because I would have no point of comparison for this.
- Before-After (BA): This is the scenario I actually implemented and is also the design used in the biocontrol study discussed at the beginning of this article. I took an inventory of my garden after I started gardening, and I reconstructed (from memory and the current state of the garden) a pre-gardening inventory of the same plot. In doing so, I was able to say with some confidence how my garden changed in terms of richness, diversity, and evenness as a result of my gardening activities.
- Control-Impact (CI): In this scenario, as with the BA scenario above, I still would be able to measure how gardening had impacted plant diversity in my garden. But instead of comparing the current state of my garden to its previous state, I would compare its current state to the current state of a control area that I had not gardened. The lawn on the other side of the fence from my garden would be a good candidate for a control, because my garden plot looked much like the adjacent area did before I started gardening. This would be feasible if I had direct access to the other side of the fence (which I don’t), but I could also eye-ball it and get estimates of plant cover for different species. Not ideal, but still feasible.
- Before-After Control-Impact (BACI): This would be a combination of BA and CI designs. Basically, I could have taken inventories of the lawn across the fence (control area) and my plot (impact area) both before and after I started gardening. I already did the BA part of it, so I would just have to add the CI aspect by also taking measurements of the lawn across the fence (or eye-balling it, as explained above).
- Randomized Controlled Trial (RCT): This is a bit harder to apply to my garden example for a couple of reasons. For one, I measured the entire population (i.e., my entire garden) rather than a sample of it. So there was in effect only one “sample”, whereas an RCT needs multiple samples per group. Secondly, the location of my garden was not randomly selected; I only had one choice for where to place my garden. But for the sake of imagination, let’s say I had a big piece of land to work with. I could divide this land up into many sections and randomly designate each section as either an impact or control section. I’d garden the impact sections and leave the control sections alone. Then I’d do inventories in both types of sections and compare diversity between them. But coming back to reality, given the land I currently have access to, this design would not be feasible.
Reflecting on my garden study
What initially started as an “After” study in my garden, with only one time point, evolved into a Before-After (BA) design with the addition of a reconstructed “Before” inventory. There were noticeable differences in plant diversity between the two time points, both qualitatively and quantitatively. And I attributed these differences to my gardening activities.
But how can I be sure that my activities caused the changes? Could the marked increase in plant diversity be due to other influences? With just the two time points, it’s technically not possible to say for sure. To rule out other possible influences, I would have needed to implement a BACI design by adding some controls in surrounding areas that are also subject to the same influences.
But practically, I am still confident that the changes I observed were largely due to my gardening interventions. The new plant species showed up because I planted them there and removed their competition (e.g., lawn grasses, weeds). All I need to do is look across the fence to see that the lawn that looked like my plot when I moved in still looks like the same lawn, whereas my plot has a bunch of new species that I planted. I think any reasonable person would agree.
So, would a before-after control impact design with randomization be ideal for many ecological experiments? Probably. But can researchers still obtain meaningful results from less-than-ideal designs? Yes. And sometimes, a less than optimal design may be the only feasible option for getting the desired answers.
AI RESPONSIBILITY RUBRIC
This rubric shows human vs AI contribution across stages of developing the article. The rubric was generated by AI and reviewed by Taylan, making adjustments as needed.
--------------------
CONCEPT/PLANNING
Human 70% | AI 30%
[==============......]
The core idea and framing originated from a real conversation and the author's own prior article series; AI assistance helped shape the outline structure for applying the concept.
--------------------
RESEARCH/VERIFICATION
Human 45% | AI 55%
[=========...........]
AI conducted literature search and retrieval of the source paper and verified specific claims; author's own critical questioning drove key clarifications and caught a conceptual gap in the source material's applicability.
--------------------
WRITING
Human 90% | AI 10%
[==================..]
Draft prose was independently written by the author, following an outline structure discussed collaboratively beforehand.
--------------------
EDITING/REFINEMENT
Human 30% | AI 80%
[======..............]
Author made a first editing pass. AI conducted logic-check, fact-check, and proofreading passes, catching grammar, punctuation, consistency, and citation issues; author reviewed and implemented corrections.
--------------------
AI Tools: Claude Sonnet 5