At Janelia Farm, the Fly EM Project is developing technology that will enable us to create a map of every neuron and synapse in the Drosophila nervous system, using novel approaches to electron microscopy (EM) as the foundation. In the same way that the fly genome paved the way for larger projects, including sequencing the human genome, Fly EM may ultimately contribute to our understanding of the human brain by establishing a fly ‘connectome’ – a map that shows how all neurons in the fly brain are connected to each other. We began our entry into EM reconstruction with the fly’s adult visual system, where much is known about cell types from previous EM and histological studies2-5, as well as ongoing studies in the Fly Light Project. In addition to establishing and publishing a fly connectome, Fly EM will make technology and methodology available that is needed to perform large-scale EM reconstructions.
- Release of web tools to explore parts of our seven-column reconstruction (connectome still being refined) (link)
- Release of one FiB-SEM medulal column of data (ground truth labels and original grayscale) accessible through our DVID API (link)
- New resources describing Fly EM reconstruction analysis and methodology (link)
- Publication of "Graph-based active learning of agglomeration (GALA): a Python library to segment 2D and 3D neuroimages" (Nunez-Iglesias, Kennedy, Plaza, Chakraborty, Katz) in Frontiers in Neuroinformatics. [April '14]
The goals of Fly EM are:
To date, serial section transmission electron microscopy (ssTEM) is the only technique to provide a proof-of-concept "connectome" by reconstructing the entire nervous system of C. elegans1, a process that took more than twelve years. (We believe that FIB-SEM is now emerging as an imaging technique that often has superior properties compared to ssTEM.) To scale up from 302 neurons in the worm to an estimated 100,000 neurons comprising the adult fly brain requires electron microscopy datasets many orders of magnitude larger and automated image processing. To meet these requirements, the Fly EM project has assembled a team of talented individuals with diverse expertise, including computer science, electrical engineering, theoretical physics, Drosophila neuroanatomy, and, of course, electron microscopy. Using the Drosophila Genome Project as its model, the Fly EM project provides the organizational and operational expertise to establish and run a large-scale pipeline for EM reconstruction. Pipelining this complex process allows Fly EM to pursue the ambitious goal of reconstructing an entire fly nervous system.
FlyEM has explored multiple strategies for large-scale imaging, in particular, serial section TEM and FIB-SEM.
To reconstruct large regions of Drosophila nervous tissue, we optimized specimen preparation (eg. high-pressure freezing followed by freeze-substitution) for automated image processing, as well as serial sectioning. The fly is amenable to large-scale EM imaging, in part, because the larval and adult nervous systems are within the range of scales suitable for serial sectioning (10^3 – 10^6 µm3). To date, we have collected over 3000 serial sections through the lamina and medulla and over 5000 serial sections through the entire L1 CNS with conventional ultramicrotomy. Imaging these sections at a resolution sufficient to detect synapses requires hundreds of thousands of images, a time consuming process without automation. We improved imaging throughput by customizing the Leginon Automated TEM Image Acquisition System (developed by NanoImaging Services, Inc. in La Jolla, CA).
Before 2012, our group acquired datasets primarily using ssTEM with anisotropic resolution (e.g., 3 x 3 x 50 nm). The relatively poor resolution in the Z direction limited our ability to trace very fine processes and the effectiveness of automatic image segmentation. Therefore, we are now refining and improving techniques using FIB-SEM (spearheaded by Harald Hess's Lab) to produce our datasets. While FIB-SEM technology has a smaller field of view when compared to ssTEM, improvements to scalability have allowed our group to generate datasets encompassing more than seven complete columns of medulla data with isotropic resolution of 10 x 10 x 10 nm and minimal imaging artifacts.
Employing methods from electrical engineering and computer vision, the Fly EM team lead by Chklovskii and Scheffer labs developed a semi-automated pipeline for serial section TEM reconstruction (Chklovskii et al. 2010), which consists of three main image processing tasks: registration, segmentation, and linkage. Briefly, registration involves horizontal alignment of images to produce a “panorama” of each EM section, and vertical alignment of each section to produce an image stack. Segmentation partitions the stack into voxels, defining profiles that belong to neurons. Linkage is the final process by which voxels belonging to putative neurons are linked across adjacent sections. Despite our ability to automate image processing and segment large datasets, the resulting reconstruction contains errors, which must be corrected in a proofreading step. We have adapted this procedure to now work with isotropic FIB-SEM datasets. While some of these techniques remain similar in spirit, there is no need for an explicit linkage step since segmentation that was previously applied separately on each section in 2D can now be applied in 3D.
The last step of the EM reconstruction pipeline is proofreading and synapse annotation, which is performed using custom written visualization and editing software called Raveler. Raveler includes tools for correcting computer segmentation errors (such as erroneously assigned voxels), annotating location of synapses and highlighting other areas of interest. To maintain performance while proofreading large datasets (> 2-3 terabytes), Raveler splits images into tiles that are dynamically loaded like Google Maps.
At the core of the proofreading process is a team of 9 proofreaders (see Gallery) who participate in a training curriculum that includes (1) proofreader training datasets, (2) proofreader instruction manual, (3) comparison software for identifying discrepancies between proofread datasets, and (4) automated tracking system for monitoring proofreading rate, as well as cataloging proofread datasets. At present, our proofreading team is comprised of 6 full-time and 2 part-time proofreaders.
Our proofreading workflow is divided by objectives. The first objective is to pull out cell shapes and identify cells of interest. The second objective is to identify and trace post-synaptic dendrites back to identified cells. To accomplish the first objective, ~ 80% of the volume must be proofread. Neuroanatomists identify relevant cell shapes and annotate the location of pre- and post-synaptic sites using tools in Raveler. To accomplish the second objective, proofreaders link the annotated post-synaptic sites to cells of interest and the final connectome is produced.
As we transition to FIB-SEM datasets, our algorithms are more capable of automatically segmenting the volume and restricting the parts that must be examined by proofreaders (focused proofreading). For example, we determine the significance and probability of potential errors in the segmentation and have proofreaders avoid areas where we are either confident of correctness or where an error minimally impacts the topology/reconstruction (Plaza, Scheffer, Saunder 2012).
- Dense reconstruction of seven medullla columns using a FIB-SEM dataset. The reconstruction took less and had better quality than our previous 1-column connectome.
- Indentification of candidate pathway for elementary motion detection (EMD) in the Drosophila. More details here.
- Dense reconstruction in a TEM dataset of cell shapes and connectome in single medulla column
- Sparse tracing of major pathways believed to contribute to the elementary motion detector in 19 medulla columns of a TEM dataset
- Reconstruction of single lamina cartridge as a proof-of-principle for our pipeline
- FIB-SEM imaging technology
- STEM technology
- Image stack of seven columns of medulla data using FIB-SEM
- Image stack of lamina cartridge and larval neuropil using FIB-SEM as proof-of-principle
- Complete series of serial sections through L1 CNS
- Algorithms for automatic segmentation and registration of EM images (2D and 3D)
- Raveler software for editing computer segmentation and synapse annotation
- Establishing proofreading team at Janelia
- Establishing off-site proofreading effort at Dalhousie University
Fly EM will generally pursue an open policy with our datasets, software, and algorithms after relevant publications. When an EM reconstruction is published, the derived connectome and reconstructed neuronal skeletons will be made available online. The raw data and annotatations will be made available upon request as logistics dictate. To encourage further collaboration and scientific discovery, a small fraction of our raw data and corresponding segmentation will be made available independent of publication. Our goal is to enable others who wish to approach the many algorithmic challenges, but who do not have access to an EM facility, to have the data they need to support methods development, as well as our results to use as a benchmark. Fly EM emphasizes publication of supporting techniques and software approaches before major EM reconstruction releases to encourage rapid feedback from the community and adoption of our strategies.
FlyEM maintains much of its software in the open-source repository GitHub:http://janelia-flyem.github.com. We will provide information on official release versions of these packages on git-hub when it reaches reasonable maturity.
Recent results have shown the possibility of both reconstructing connectomes of small but biologically interesting circuits and extracting from these connectomes insights into their function. However, these reconstructions were heroic proof-of-concept experiments, requiring person-months of effort per neuron reconstructed, and will not scale to larger circuits, much less the brains of entire animals. In this paper we examine what will be required to generate and use substantially larger connectomes, finding five areas that need increased attention: firstly, imaging better suited to automatic reconstruction, with excellent z-resolution; secondly, automatic detection, validation, and measurement of synapses; thirdly, reconstruction methods that keep and use uncertainty metrics for every object, from initial images, through segmentation, reconstruction, and connectome queries; fourthly, processes that are fully incremental, so that the connectome may be used before it is fully complete; and finally, better tools for analysis of connectomes, once they are obtained.
The aim in high-resolution connectomics is to reconstruct complete neuronal connectivity in a tissue. Currently, the only technology capable of resolving the smallest neuronal processes is electron microscopy (EM). Thus, a common approach to network reconstruction is to perform (error-prone) automatic segmentation of EM images, followed by manual proofreading by experts to fix errors. We have developed an algorithm and software library to not only improve the accuracy of the initial automatic segmentation, but also point out the image coordinates where it is likely to have made errors. Our software, called gala (graph-based active learning of agglomeration), improves the state of the art in agglomerative image segmentation. It is implemented in Python and makes extensive use of the scientific Python stack (numpy, scipy, networkx, scikit-learn, scikit-image, and others). We present here the software architecture of the gala library, and discuss several designs that we consider would be generally useful for other segmentation packages. We also discuss the current limitations of the gala library and how we intend to address them.
We aim to improve segmentation through the use of machine learning tools during region agglomeration. We propose an active learning approach for performing hierarchical agglomerative segmentation from superpixels. Our method combines multiple features at all scales of the agglomerative process, works for data with an arbitrary number of dimensions, and scales to very large datasets. We advocate the use of variation of information to measure segmentation accuracy, particularly in 3D electron microscopy (EM) images of neural tissue, and using this metric demonstrate an improvement over competing algorithms in EM and natural images.
A visual motion detection circuit suggested by Drosophila connectomicsNature 2013
S. Takemura, A. Bharioke, Z. Lu, A. Nern, S. Vitaladevuni, P. K. Rivlin, W. T. Katz, D. J. Olbris, S. M. Plaza, P. Winston, T. Zhao, J. Horne, R. D. Fetter, S. Takemura, K. Blazek, L. Chang, O. Ogundeyi, M. A. Saunders, V. Shapiro, C. Sigmund, G. M. Rubin, L. K. Scheffer, I. A. Meinertzhagen, and D. B. Chklovskii Nature, 500:175–181 (2013)
Animal behaviour arises from computations in neuronal circuits, but our understanding of these computations has been frustrated by the lack of detailed synaptic connection maps, or connectomes. For example, despite intensive investigations over half a century, the neuronal implementation of local motion detection in the insect visual system remains elusive. Here we develop a semi-automated pipeline using electron microscopy to reconstruct a connectome, containing 379 neurons and 8,637 chemical synaptic contacts, within the Drosophila optic medulla. By matching reconstructed neurons to examples from light microscopy, we assigned neurons to cell types and assembled a connectome of the repeating module of the medulla. Within this module, we identified cell types constituting a motion detection circuit, and showed that the connections onto individual motion-sensitive neurons in this circuit were consistent with their direction selectivity. Our results identify cellular targets for future functional investigations, and demonstrate that connectomes can provide key insights into neuronal computations.
The ability to automatically segment an image into distinct regions is a critical aspect in many visual processing applications. Because inaccuracies often exist in automatic segmentation, manual segmentation is necessary in some application domains to correct mistakes, such as required in the reconstruction of neuronal processes from microscopic images. The goal of the automated segmentation tool is traditionally to produce the highest-quality segmentation, where quality is measured by the similarity to actual ground truth, so as to minimize the volume of manual correction necessary. Manual correction is generally orders-of-magnitude more time consuming than automated segmentation, often making handling large images intractable. Therefore, we propose a more relevant goal: minimizing the turn-around time of automated/manual segmentation while attaining a level of similarity with ground truth. It is not always necessary to inspect every aspect of an image to generate a useful segmentation. As such, we propose a strategy to guide manual segmentation to the most uncertain parts of segmentation. Our contributions include 1) a probabilistic measure that evaluates segmentation without ground truth and 2) a methodology that leverages these probabilistic measures to significantly reduce manual correction while maintaining segmentation quality.
Reconstructing neuronal circuits at the level of synapses is a central problem in neuroscience, and the focus of the nascent field of connectomics. Previously used to reconstruct the C. elegans wiring diagram, serial-section transmission electron microscopy (ssTEM) is a proven technique for the task. However, to reconstruct more complex circuits, ssTEM will require the automation of image processing. We review progress in the processing of electron microscopy images and, in particular, a semi-automated reconstruction pipeline deployed at Janelia Farm. Drosophila circuits underlying identified behaviors are being reconstructed in the pipeline with the goal of generating a complete Drosophila connectome.
Increasing depth resolution of electron microscopy of neural circuits using sparse tomographic reconstructionComputer Vision and Pattern Recognition (CVPR) 2010
A. Veeraraghavan, A. V. Genkin, S. Vitaladevuni, L. Scheffer, S. Xu, H. Hess, R. Fetter, M. Cantoni, G. Knott, and D. B. Chklovskii Computer Vision and Pattern Recognition (CVPR), (2010)
Team Members Groups