February 2011 - January 2014
Electron microscopy (EM) is still the best technique for producing data from which one can unambiguously determine the complete synaptic connectivity of neuronal assemblies.
Although recent innovations have made imaging more efficient, the subsequent step of image analysis is a persistent bottleneck. Many years of human effort solved the image analysis problem for c. elegans, but this approach is too costly to enable the routine and rapid production of wiring diagrams - especially for circuits with tens or hundreds of thousands of neurons. Automation of image analysis is therefore a critical problem for neurobiology, and the one that forms the core of my research program.
Research
Neural wiring in a rabbit retina reconstructed using serial block-face scanning electron microscopy and automated image analysis algorithms. Each individually colored object denotes a putatively distinct physical wire identified by an algorithm." with credits as "Viren Jain (Janelia), Sebastian Seung & Srinivas Turaga (MIT), Kevin Briggman & Moritz Helmstaedter & Winfried Denk (Max Planck Institute)
Machine Learning of Image Analysis
To infer a connectivity matrix from images of dense neuropil, two image analysis tasks must be performed: (1) identifying synapses, and (2) tracing neurites to their parent cell bodies. The first task can be posed as a visual object recognition problem, in which the goal is to determine whether an image contains an example of an object category.
The second task can be posed as an image segmentation problem, in which the goal is to group pixels into distinct partitions that correspond to physical objects. In order to produce useful wiring diagrams, image segmentation must be performed with extraordinary accuracy. A mistake in synapse identification will cause at most a single error in a connectivity matrix, but a single error in neurite tracing could misassign thousands of synapses.
In previous work, we have introduced new image segmentation algorithms based on a machine learning approach to image analysis. By developing novel classifiers and learning algorithms specialized for the problem of segmentation, our approach has yielded new levels of performance in automated EM reconstruction. However, these methods have largely focused on a relatively local analysis of the image volume; further improvements in accuracy are likely to require automated methods that can reason about much larger amounts of image context. We are pursuing such improvements through research in several directions:
- Segmentation is an example of a structured prediction problem. Probabilistic graphical models and non-probabilistic energy-based models are powerful representations for dealing with structured prediction problems, but generally define very difficult learning and inference problems. For EM reconstruction we will eventually encounter models with millions of objects and a practically infinite combinatorial space of configurations of those objects. What learning and inference algorithms are effective for dealing with such problems? There is a large and growing literature on efficient (approximate) methods for inference and learning in such models, but such research has been done on different domains and problems. EM reconstruction may require a fresh approach to these issues.
- Neurites are very specialized 3d structures and therefore there should exist effective low-dimensional features that describe them. What is the best approach for devising such descriptors (e.g., unsupervised learning, human specification, or end-to-end discriminative learning) and how can they be used to best improve EM reconstruction accuracy?
Semi-automated reconstruction by integrating machine learning, computer vision, and human-computer interaction
In addition to advancing the basic computer vision and machine learning research involved in automated reconstruction, we are solving the science and engineering challenges involved in capitalizing on existing automated techniques in reconstructing wiring diagrams. In particular, using state of the art but imperfect automated techniques to produce accurate descriptions of connectivity requires sophisticated software that enables humans to ‘proofread’ computer reconstructions by identifying and correcting errors. Our approach integrates machine learning algorithms directly into the proofreading process, which will enable progress on several important problems:
- The true metric of interest in a reconstruction project is the amount of human effort required to proofread and correct errors in an automated segmentation (the ‘nuisance metric’). However, all machine learning approaches to image segmentation are designed to optimize some other metric (such as pixel error). Can a machine learning strategy be devised that directly optimizes the nuisance metric?
- Machine learning requires ground truth, which in EM reconstruction is typically provided by humans and is thus expensive to acquire. How can human input be selectively acquired such that it is most informative for a particular learning algorithm? The general form of this problem has been well-studied as ‘active learning,’ but has yet to be rigorously pursued within the context of image segmentation. A successful solution to this problem will increase performance of automated reconstruction by maximizing the information content provided by human labeling.
Reconstructing Sensory Circuits
In collaboration with labs at Janelia and elsewhere, we are applying our reconstruction algorithms and tools to answer fundamental biological questions related to how local circuitry in sensory systems process information. We are excited about combining measurements of neuronal activity and connectivity to try and understand computational principles that govern the nervous system.