MOSAIC: Multimodal In Vivo Imaging Data Powers AI Models for Living Systems

In a new study published in Nature Methods titled, “A multimodal adaptive optical microscope for in vivo imaging from molecules to organisms,” researchers from University of California, Berkeley present high-powered microscopes that can track the development of live specimens, including cell movement through tissue, the evolution of internal cellular structures, and shuttling of proteins and other molecules within the cell. The system, named Multimodal Optical Scope with Adaptive Imaging Correction (MOSAIC), has been implemented in more than a dozen worldwide labs over the past six years. 

“Life has to be studied in living tissue, holistically, and over fast timescales and for long periods of time,” said Eric Betzig, PhD, professor of molecular and cell biology at UC Berkeley, 2014 Nobel Prize in Chemistry, and co-corresponding author on the study. “You can’t study something as complex as a cell or organism just by looking at the parts individually—there are something like 40 million protein molecules alone of 20,000 different types.” 

The microscope uses a large “vision” language model (LVLM), similar to ChatGPT, to measure petabytes of data, the equivalent of about 500 billion pages of text.  

Betzig, who is also a Howard Hughes Medical Institute (HHMI) investigator, refers to the imaging data as five-dimensional (5D) composed of three spatial dimensions, plus time and color. The color comes from fluorescent labels that allow scientists to track multiple subcellular structures simultaneously, such as organelles, membranes, the cytoskeleton and more, as they migrate, change shape, divide and interact over time. 

In one video, the authors capture a zebrafish regrowing its tail fin. The video revealed tiny events inside living tissue that are normally difficult to visualize, such as cells near the wound releasing small communication packets, microscopic fibers beneath the skin shifting as the tissue repaired itself, two repair cells fusing together and a red blood cell briefly getting trapped as new blood vessels were remodeled. 

Ian Swinburne, PhD, assistant professor of molecular and cell biology at UC Berkeley and collaborator on the work, emphasizes that there’s a wealth of information in these large movies across scales, but it can be difficult for a very well-trained biologist to interrogate the data.

“AI can help us interface with the data and ask or answer questions more easily. Like, ‘How many macrophages are crawling into my tissue during an infection?’ or ‘Can I predict when a cell’s going to start leaving its organ?’ That happens in development but also in cancer during metastasis,” said Swinburne. 

Building an LVLM or AI that can handle petabytes of imaging data is a main focus of Berkeley’s Advanced Bioimaging Center, which hopes to create a first-of-its-kind Cell Observatory. 

“The impact of MOSAIC will be minimal until we build an AI model that can deal with the data that comes out of those systems. We basically have a gold mine, but we have no ability to get the gold out,” said Srigokul “Gokul” Upadhyayula, PhD, assistant professor in residence of cell biology, development and physiology at UC Berkeley. “The primary output of our Cell Observatory Initiative will be an AI mind that’s able to be our scientific partner in extracting these observations.”