Branson Lab

Nervous systems are complex. Even the simple fruit fly's nervous system contains hundreds of thousands of neurons, making millions of connections with each other. The output of these computations, the animal's behavior, is highly diverse and equally complex. To understand these systems, we need to collect, analyze, and interpret large data sets detailing the anatomy and activity of populations of neurons and their contributions to the animal's behavior. Technological advances are enabling the collection of such data, usually in the form of images and videos. Our lab's research is focused on how to extract scientific understanding from these large data sets. We:

• Develop new machine vision and learning algorithms tailored to the types of data and information available in large-scale, technology-enabled neuroscience experiments. 
• Address new types of questions about the brain and behavior only approachable with the new data analysis techniques we are developing.
• Develop robust, general-purpose implementations of these algorithms usable by biologists that will have an impact on neuroscience research outside our lab.

Quantifying animal behavior

Our analyses begin with video of the animals behaving. This video captures a large amount of information about the behavior of the animals, but is high-dimensional and cannot be interpreted quantitatively in its raw form. How should we represent the animals' behaviors? Should we discretize behavior into categories, or should we use a more continuous representation? What is the behavioral "vocabulary" of an animal? In our research, we are searching for scientifically meaningful, concise yet detailed, quantitative representations of animal behavior. 

Left: Using Ctrax, we can track the position, orientation, and wing angles of groups of interacting flies. Right: Using cascaded pose regression, we can track the paw of a mouse in three dimensions.

We use machine vision and learning to answer these questions. We are developing machine vision software to automatically track animals -- to estimate their position and pose in every frame of an input video. We are also developing machine learning software to automatically categorize their behaviors in each frame of the video. JAABA uses a supervised machine learning paradigm to learn behavior classifiers that can automatically reproduce biologists' behavior definitions. In the supervised framework, biologists define behaviors by example. They manually annotate the behaviors the animals are performing in a small subset of the video frames, and the machine learning algorithm searches for the parametric function that can reproduce these annotations automatically. 

Results of applying 15 automatic fly behavior classifiers trained using JAABA.

We are currently exploring weakly supervised machine learning approaches to automatically discover new behavioral phenotypes. Can we record signals other than the input video that are informative of the animals' behavioral states (e.g. neural activity via calcium imaging), and use these as a form of weak, noisy label? Can we combine statistical analyses of unlabeled data with annotations from the human to better explore large data sets and discover structure?

New paradigms for machine vision and learning in practice

It is an exciting time for machine vision and learning -- algorithms are becoming accurate and efficient enough that they can be used to solve real problems. How do we take machine vision and learning algorithms out of the computer science lab and make them usable? We have found that the standard supervised machine learning framework fails in many biology applications. Ideally, biologists and computer scientists could work together, with biologists creating annotated data sets indicating the problem and desired solution, and computer scientists developing algorithms to automatically reproduce these manual annotations. Unfortunately, such a clean division of labor is usually not possible. Scientific research is by nature an iterative process, and it is often difficult a priori to formulate a mathematical criterion to be optimized. 

Screencast of beginning to train a wing grooming classifier using JAABA.

Animal behavior annotation is a prime example of this. The behavioral "vocabulary" of animals is unknown. For a specific behavior category, it is difficult to draw the boundaries of that category (e.g. how many steps backwards must a fly take to be considered "backing up"?). Second, the costs of different types of errors are different, and unknown. Third, the data distribution may vary from experiment to experiment as we perform different manipulations on the animals.

To address these issues, we are exploring new paradigms for machine learning. We are developing interactive systems in which the user can iteratively add new annotations, run machine learning to find new structure in the data (train new classifiers, cluster the data, etc.), then show this structure to the user. We are also exploring modalities beyond per-example labels for communicating the user's intuition to the machine learning system, such as triplet-based distance relationships. Finally, we are developing new data visualization techniques for displaying the machine learning algorithm's current estimates of structure to the user.

Using machine vision and learning to gain scientific understanding

We are using the tools we are developing to gain insight into nervous system function, evolution, and ethology. We use these tools to answer a wide variety of questions, such as:

• What populations of neurons are involved in a behavior? How do they work together to generate and control this behavior?
• What behavioral differences have evolved between closely related species? Why? 
• How does an animal's history (in particular social history) affect its behavior? How is this state stored?
• What are the component behaviors of social interactions in animals? What are the stimulus-response relationships between these components? 

Besides providing insight into basic questions in the life sciences, performing large-scale experiments to answer this diverse set of questions informs us as to the limitations of current techniques, and what new tools will have the biggest, broadest impact. Each of these experiments provides a weak signal of the "language" of behavior for Drosophila, and what methodologies are useful in illuminating this underlying structure.