Cryo-EM single particle analysis can solve multiple protein structures contained in a sample by classification. However, the information on the dynamics between the solved structures is lost and it can only be inferred. About this problem, cryoDRGN, a deep learning approach for solving the three-dimensional reconstruction and structural classification that was announced in 2020, breaks away from discrete data partition. The method is based on an auto-encoder, and realizes continuous structural classification by constructing a latent space that separates the information depending on the projection parameters from the input particle image. In this paper, we explain conventional classification method in single particle analysis and deep learning topics that are the background of cryoDRGN. Then, as a benchmark for structural classification, we try three-dimensional reconstruction on the actual data of GroEL/ES having 6 kinds of complexes.

Examination of microscopic images is a fundamental method in cell biology. Traditionally, the use of microscopic images has tended to be limited to qualitative interpretation based on visual inspections. However, this conventional method may lack objectivity, and the human labor cost becomes huge when a large number of image datasets need to be examined. To overcome these problems, we have been developing image analysis frameworks for quantitative evaluation and classification of cell features using machine learning techniques. In this article, we review our work, including a microscopic data mining method based on quantitative evaluation and clustering of cytoskeletal structures, the CARTA framework for versatile biomedical image classification, and a semi-automatic method to detect intracellular structures from wide-area microscopic images. We also discuss the benefits of image analysis and machine learning from the perspective of an experimental biologist.

Single particle analysis using cryogenic electron microscopy has been rapidly developing as a high-resolution structure determination method for biological macromolecules. In particular, it has enabled direct visualization of macromolecules, to which other structure determination methods could not be applied due to size, structural heterogeneity and compositional variability of the structures. Since single particle analysis is an image-processing-heavy method, a wide variety of algorithms have been tried and many procedures have already been automated, and so the procedure is steadily becoming more routine work. However, until now, only general abilities of human visual recognition and comprehensive judgment, which are based on past experiences of many structural analysis practices, have been unable to be automated. However, deep learning techniques, which have revolutionized the AI field in recent years, are changing this situation. This article will give an overview of deep learning applications in various processing steps of single particle analysis, and introduce crYOLO and Topaz in more details as two best representatives of deep-learning-based particle picking that have reached a practical level of full automation.

Recent years have seen a rapid expansion in the field of electron microscopic-based (EM) connectomics, which targets 3D reconstruction of neuronal networks from an EM volumetric image. The EM connectomics has attracted attention, because this is only the way to obtain actual and overall neuronal connectivity. In the EM connectomics, image processing is an important challenging topic as well as image acquisition itself. In particular, the process called “image segmentation” has been accelerated owing to deep convolutional neural networks, enabling the study of large-scale connectomes, such as the whole fly brain. We would like to review such information technologies, as well as a unifying software, UNI-EM, and database technologies for the EM connectiomics.

A virus is a microscopic organism, which sometimes causes disease to the host. There are many pathogenic human viruses with RNA genomes such as influenza, measles, Ebola, rabies, corona, and West Nile viruses. Understanding virus structures is essential to clarify their replication mechanism and to develop anti-viral drugs. However, structural analysis of RNA viruses is challenging because most of them have a flexible and fragile nature. Here, I will describe our structural studies using electron microscopies on RNA viruses such as influenza and Ebola viruses.

Pulsed electron wave-packet had been applied to a transmission electron microscopy (TEM) to improve the time-resolution. The pulsed electrons are created by photoemission process using a combination of a photocathode material and pulsed laser. We review time-scales of phenomena, performance of photocathodes and recent researches of time-resolved TEM. Especially the energy and momentum spreads of photocathode is discussed by our experimental result of time-resolved TEM using an NEA semiconductor photocathode. To introduce a novel electron microscopy using a quantum effect, we explain quantum physics in TEM involving special relativity, Dirac equation and a notation of two component spinor. Finally, we discuss key technologies and significant parameters for high temporal and high spatial resolutions.

Digital camera is the key instrument for modern electron microscopy. Different from traditional negative films, it enables online data evaluation and feedback to the microscope. For example, automated alignment requires images as an input data from misaligned and aligned conditions, and automated data acquisition needs corrections for feature tracking, focusing, and related alignment. By the recent development in the sensor technology, we are now migrating from CCD based camera to CMOS based camera. CMOS sensor has a capability to speed up the frame rate without sacrificing image resolution. This feature opens a way to the application requiring time resolution like in-situ imaging. And now the direct detection detector (DDD) technology expands the application field from cryo electron microscopy to material science field. The key performance of DDD enhances by electron counting processing. It eliminates the image blurring caused by point spread function and unwanted noises. The technology is now applying in in-situ imaging and electron energy loss spectroscopy.

Analyzing the structure of plants is important to understand their functions. In recent years, it has been known that immobile plants respond rapidly to environmental changes by responding to external stimuli received in leaves and roots by transmitting signal molecules throughout the body with cell-cell movements. Therefore, it is expected to be observed microscopically at the tissue level, organ level, and finally, the whole body at the cellular resolution. However, to observe the deep inside of a plant body with fluorescence microscopy, various difficulties such as autofluorescence of plants are confronted. In this article, I introduce the development of microscopy and sample preparation techniques to overcome the difficulties.

NMR crystallography, in which diffraction and solid-state NMR are combinedly used, is introduced. As the information from diffraction and solid-state NMR are complementary, NMR crystallography approach provides comprehensive information on crystalline structure and is especially useful to locate hydrogen atom at a right position. While X-ray has been used in most cases, here, we introduce microED in the NMR crystallography framework. This approach enables the crystalline structure solution including very accurate hydrogen positions from nano- to micro-crystalls. The applications to salt/co-crystal discrimination in pharmaceutical sciences and to structural elucidation of cimetidine form-B whose structure is previously unknown are demonstrated.