We aim to improve segmentation through the use of machine learning tools during region agglomeration. We propose an active learning approach for performing hierarchical agglomerative segmentation from superpixels. Our method combines multiple features at all scales of the agglomerative process, works for data with an arbitrary number of dimensions, and scales to very large datasets. We advocate the use of variation of information to measure segmentation accuracy, particularly in 3D electron microscopy (EM) images of neural tissue, and using this metric demonstrate an improvement over competing algorithms in EM and natural images.

We present an approach to solving computer vision problems in which the goal is to produce a high-dimensional, pixel-based interpretation of some aspect of the underlying structure of an image. Such tasks have traditionally been categorized as “low-level vision” problems, and examples include image denoising, boundary detection, and motion estimation. Our approach is characterized by two main elements, both of which represent a departure from previous work. The first is a focus on convolutional networks, a machine learning strategy that operates directly on an input image with no use of hand-designed features and employs many thousands of free parameters that are learned from data. Previous work in low-level vision has been largely focused on completely handdesigned algorithms or learning methods with a hand-designed feature space. We demonstrate that a learning approach with high model complexity, but zero prior knowledge about any specific image domain, can outperform existing techniques even in the challenging area of natural image processing. We also present results that establish how convolutional networks are closely related to Markov random fields (MRFs), a popular probabilistic approach to image analysis, but can in practice can achieve significantly greater model complexity. The second aspect of our approach is the use of domain specific cost functions and learning algorithms that reflect the structured nature of certain prediction problems in image analysis. In particular, we show how concepts from digital topology can be used in the context of boundary detection to both evaluate and optimize the high-order property of topological accuracy. We demonstrate that these techniques can significantly improve the machine learning approach and outperform state of the art boundary detection and segmentation methods. Throughout our work we maintain a special interest and focus on application of our methods to connectomics, an emerging scientific discipline that seeks highthroughput methods for recovering neural connectivity data from brains. This application requires solving low-level image analysis problems on a tera-voxel or peta-voxel scale, and therefore represents an extremely challenging and exciting arena for the development of computer vision methods.

Insects constitute the most species-rich radiation of metazoa, a success that is due to the evolution of active flight. Unlike pterosaurs, birds and bats, the wings of insects did not evolve from legs, but are novel structures that are attached to the body via a biomechanically complex hinge that transforms tiny, high-frequency oscillations of specialized power muscles into the sweeping back-and-forth motion of the wings. The hinge consists of a system of tiny, hardened structures called sclerites that are interconnected to one another via flexible joints and regulated by the activity of specialized control muscles. Here we imaged the activity of these muscles in a fly using a genetically encoded calcium indicator, while simultaneously tracking the three-dimensional motion of the wings with high-speed cameras. Using machine learning, we created a convolutional neural network that accurately predicts wing motion from the activity of the steering muscles, and an encoder-decoder that predicts the role of the individual sclerites on wing motion. By replaying patterns of wing motion on a dynamically scaled robotic fly, we quantified the effects of steering muscle activity on aerodynamic forces. A physics-based simulation incorporating our hinge model generates flight manoeuvres that are remarkably similar to those of free-flying flies. This integrative, multi-disciplinary approach reveals the mechanical control logic of the insect wing hinge, arguably among the most sophisticated and evolutionarily important skeletal structures in the natural world.

Recent developments in machine vision methods for automatic, quantitative analysis of social behavior have immensely improved both the scale and level of resolution with which we can dissect interactions between members of the same species. In this paper, we review these methods, with a particular focus on how biologists can apply them to their own work. We discuss several components of machine vision-based analyses: methods to record high-quality video for automated analyses, video-based tracking algorithms for estimating the positions of interacting animals, and machine learning methods for recognizing patterns of interactions. These methods are extremely general in their applicability, and we review a subset of successful applications of them to biological questions in several model systems with very different types of social behaviors.

Connections between neurons can be found by checking whether synapses exist at points of contact, which in turn are determined by neural shapes. Finding these shapes is a special case of image segmentation, which is laborious for humans and would ideally be performed by computers. New metrics properly quantify the performance of a computer algorithm using its disagreement with ’true’ segmentations of example images. New machine learning methods search for segmentation algorithms that minimize such metrics. These advances have reduced computer errors dramatically. It should now be faster for a human to correct the remaining errors than to segment an image manually. Further reductions in human effort are expected, and crucial for finding connectomes more complex than that of Caenorhabditis elegans.

Pathogen-mediated activation of macrophages arms innate immune responses that include enhanced surface ruffling and macropinocytosis for environmental sampling and receptor internalization and signaling. Activation of macrophages with bacterial lipopolysaccharide (LPS) generates prominent dorsal ruffles, which are precursors for macropinosomes. Very rapid, high-resolution imaging of live macrophages with lattice light sheet microscopy (LLSM) reveals new features and actions of dorsal ruffles, which redefine the process of macropinosome formation and closure. We offer a new model in which ruffles are erected and supported by F-actin tent poles that cross over and twist to constrict the forming macropinosomes. This process allows for formation of large macropinosomes induced by LPS. We further describe the enrichment of active Rab13 on tent pole ruffles and show that CRISPR deletion of Rab13 results in aberrant tent pole ruffles and blocks the formation of large LPS-induced macropinosomes. Based on the exquisite temporal and spatial resolution of LLSM, we can redefine the ruffling and macropinosome processes that underpin innate immune responses.

Macroscale fluorescence imaging is increasingly used to observe biological samples. However, it may suffer from spectral interferences that originate from ambient light or autofluorescence of the sample or its support. In this manuscript, we built a simple and inexpensive fluorescence macroscope, which has been used to evaluate the performance of Speed OPIOM (Out of Phase Imaging after Optical Modulation), which is a reference-free dynamic contrast protocol, to selectively image reversibly photoswitchable fluorophores as labels against detrimental autofluorescence and ambient light. By tuning the intensity and radial frequency of the modulated illumination to the Speed OPIOM resonance and adopting a phase-sensitive detection scheme that ensures noise rejection, we enhanced the sensitivity and the signal-to-noise ratio for fluorescence detection in blot assays by factors of 50 and 10, respectively, over direct fluorescence observation under constant illumination. Then, we overcame the strong autofluorescence of growth media that are currently used in microbiology and realized multiplexed fluorescence observation of colonies of spectrally similar fluorescent bacteria with a unique configuration of excitation and emission wavelengths. Finally, we easily discriminated fluorescent labels from the autofluorescent and reflective background in labeled leaves, even under the interference of incident light at intensities that are comparable to sunlight. The proposed approach is expected to find multiple applications, from biological assays to outdoor observations, in fluorescence macroimaging.

A 60-year-old man diagnosed with macular telangiectasia type 1 (MacTel 1) was treated for 3 years with monthly aflibercept (Eylea; Regeneron, Tarrytown, NY) and serially imaged with spectral-domain optical coherence tomography. When administered monthly, aflibercept appeared to have a beneficial effect on macular edema secondary to MacTel 1. Visual acuity preservation despite minimal chronic macular edema could be attributed to the lack of significant photoreceptor disruption.

The precise neural mechanisms within the brain that contribute to the remarkable lifetime persistence of memory remain unknown. Existing techniques to record neurons in animals are either unsuitable for longitudinal recording from the same cells or make it difficult for animals to express their full naturalistic behavioral repertoire. We present a magnetic voluntary head-fixation system that provides stable optical access to the brain during complex behavior. Compared to previous systems that used mechanical restraint, there are no moving parts and animals can engage and disengage entirely at will. This system is failsafe, easy for animals to use and reliable enough to allow long-term experiments to be routinely performed. Together with a novel two-photon fluorescence collection scheme that increases two-photon signal and a transgenic rat line that stably expresses the calcium sensor GCaMP6f in dorsal CA1, we are able to track and record activity from the same hippocampal neurons, during behavior, over a large fraction of animals’ lives.

PURPOSE: To develop switchable and tunable labels with high contrast ratio for MRI using magnetocaloric materials that have sharp first-order magnetic phase transitions at physiological temperatures and typical MRI magnetic field strengths.

METHODS: A prototypical magnetocaloric material iron-rhodium (FeRh) was prepared by melt mixing, high-temperature annealing, and ice-water quenching. Temperature- and magnetic field-dependent magnetization measurements of wire-cut FeRh samples were performed on a vibrating sample magnetometer. Temperature-dependent MRI of FeRh samples was performed on a 4.7T MRI.

RESULTS: Temperature-dependent MRI clearly demonstrated image contrast changes due to the sharp magnetic state transition of the FeRh samples in the MRI magnetic field (4.7T) and at a physiologically relevant temperature (~37°C).

CONCLUSION: A magnetocaloric material, FeRh, was demonstrated to act as a high contrast ratio switchable MRI contrast agent due to its sharp first-order magnetic phase transition in the DC magnetic field of MRI and at physiologically relevant temperatures. A wide range of magnetocaloric materials are available that can be tuned by materials science techniques to optimize their response under MRI-appropriate conditions and be controllably switched in situ with temperature, magnetic field, or a combination of both.