Selected paper: Charles, Song, Tank et al., Neural Anatomy and Optical Microscopy Simulation (NAOMi) for evaluating calcium imaging methods, bioRxiv (2019).

What is the paper about? Calcium imaging is a central method to observe neuronal activity in the brain of animal models. Many labs use rather complicated algorithms to extract meaningful information from these imaging data, but the results of those algorithms are often hard to judge: is this an artifact of the algorithm or the imaging method, or is it a biologically meaningful signal of a single neuron? The paper by Charles et al. addresses this question by simulating both neuronal activity in anatomically realistic neurons and the imaging process that makes these activity patterns visible. By knowing both the true simulated biological signals and the simulated imaging data, the procedure can benchmark analysis algorithms against a known ground truth. In addition, the method can be used to evaluate imaging modalities and their impact on resolution, movement artifacts and other factors for a specific use case.

More details: Neurons are simulated as realistic 3D structures with dendrites and axons, based on 3D EM and light microscopy data of neuronal tissue. Their activity is simulated as spikes. The spikes are transformed into slower calcium events, which in turn are transformed by the binding kinetics of the calcium indicator. Binding to the calcium indicator gives rise to a change in fluorescence, based on the Hill model of cooperative binding. Then, things are brought together by simulating the excitation laser beam and the light emitted by the fluorescent calcium sensors, assuming a typical signal to noise level and shot noise induced by the limited number of photons seen by the microscope. Altogether, this is a quite impressive simulation pipeline, covering anatomy, calcium sensors and binding kinetics as well as light simulation and relevant noise induced by instruments and other variables (for example, residual artifacts induced by the motion of the animal).

Evaluating demixing algorithms: In my opinion, the most interesting part of the paper is section 2.3 (“Evaluation of automated segmentation”). Here, the authors use their simulations to benchmark three commonly used algorithms for automated source extraction. These algorithms are used to automatically extract regions of interest (ROIs) from an imaging dataset, and are often used as a repeatable and hopefully more objective and more reliable replacement of the manual drawing of ROIs by a human being. Unfortunately, the authors do not benchmark those methods also against a human expert who manually selects ROIs.
Anyway, with respect to the three state-of-the-art algorithms, the evaluation using the calcium imaging simulations reveals a rather limited precision of all investigated methods. First, the different algorithms find rather different sets of neuronal activity units with relatively little overlap between the units found by each algorithm. In addition, also the absolute number of units found by each algorithm is quite variable (for a specific simulation of L2/3 imaging in visual cortex, the number of found components varies between 265 and 1091 for the different algorithms). More details can be found in the relevant section in the paper.
These are interesting findings. They do not disqualify demixing algorithms, but they should lower our trust that the large majority of the extracted components is based on activity signals of single neurons.

The devil’s advocate: The strength of the simulation in this paper (its realism) is also its weakness. The authors themselves state:

“Simulation-based approaches, however, often suffer from being either too simple or too complex.”

How can we be sure that they have found exactly the right level of detail? We can’t, or not very easily. There is an endless list of possible details which could have been omitted: For example, the calcium indicator concentration was set to 10 μM for the simulations, but these values might differ between neuronal types and depend on other factors as well. To give another example, the simulation assumes that the calcium concentration is constant across the dendritic tree, therefore omitting the known effect of localized calcium dynamics in dendritic spines.
And there are many more approximations (optics, physiology, anatomy) made by the authors that are difficult to judge. For example, I would have been glad to see a simulated excitation PSF at a certain tissue depth alongside with a measured PSF at the same depth (e.g., using beads injected into cortex), to make sure that the scattering simulation, which is too complex to be judged without investing a lot of time, is realistic or not. – However, these are details, and for most of the analyses performed by the paper (like the discussion of demixing algorithms), this level of detail of the simulation is probably not important – and I’m looking really forward to having this simulation tool publicly available. If it is user-friendly and easy to adapt, it could become a standard tool to check in silico what to expect for in vivo experiments.

Conclusion: Very interesting and possibly useful work, although it is difficult to understand the limitations and details of the simulation.

Further reading: A paper from he same lab on a related topic: Gauthier, J. L., Tank, D. W., Pillow, J. W. & Charles, A. S. Detecting and Correcting False Transients in Calcium Time-trace Inference.

Other paper reviews on this blog: on precise balance in the hippocampus [1]; on L5 apical dendrites [2].