Removing the parachuting artifact using two-way scanning data in high-speed atomic force microscopy

Abstract

The high-speed atomic force microscopy (HS-AFM) is a unique and prominent method to observe structural dynamics of biomolecules at single molecule level at near-physiological condition. To achieve high temporal resolution, the probe tip scans the stage at high speed which can cause the so-called parachuting artifact in the HS-AFM images. Here, we develop a computational method to detect and remove the parachuting artifact in HS-AFM images using the two-way scanning data. To merge the two-way scanning images, we employed a method to infer the piezo hysteresis effect and to align the forward- and backward-scanning images. We then tested our method for HS-AFM videos of actin filaments, molecular chaperone, and duplex DNA. Together, our method can remove the parachuting artifact from the raw HS-AFM video containing two-way scanning data and make the processed video free from the parachuting artifact. The method is general and fast so that it can easily be applied to any HS-AFM videos with two-way scanning data.

Significance

For high temporal resolution in high-speed atomic force microscopy (HS-AFM) measurements, the probe tip scans the stage at high speed which can cause the parachuting artifact. We develop a computational method to detect and remove the parachuting artifact in HS-AFM images using the two-way scanning data. The developed method is general and fast, and thus can easily be applied to any HS-AFM videos with two-way scanning data.

Introduction

Observing biomolecular structures and dynamics has long been of one of the central issues in molecular biology. Nowadays, atomic-resolution structure information can be commonly obtained by a few methods including X-ray crystallography and cryo-electron microscopy [1,2]. However, these methods primarily provide information on static snapshot structures, but not on dynamic motions. On the one hand, fluorescence measurements combined with the super-resolution methods can directly observe time-dependent molecules dynamics [3–5], but their spatial resolution is limited. Therefore, to gain spatiotemporal information at high resolution is still challenging.

In this respect, the high-speed atomic force microscopy (HS-AFM) is a unique and powerful imaging method that can observe structural dynamic of single biomolecules at near physiological solution condition [6,7]. In the HS-AFM measurement, the probe tip scans the stage surface (assumed as the xy-plane) on which target biomolecules are bound. While scanning, the probe tip moves up and down searching a certain interaction strength with the specimen. The probe tip height (z-coordinate) at which the tip detects a predefined interaction strength is recorded as the measured data. Due to much of technology development, the current state-of-the-art HS-AFM has the lateral- and vertical- spatial resolution of ~20Å and ~1.5Å, respectively, and the time resolution of 50~100 ms [7,8]. With these resolutions, walking motions of myosin V [9], power-stroke motions of dynein-c [10], and many other biomolecular structural dynamics [11–13] were successfully observed by the HS-AFM. Albeit prominent, the time resolution of the current HS-AFM, ~100 ms, is not always sufficiently high to observe biomolecular motions in real time. For example, while the probe scans an area of interest once, target molecules may move markedly, which results in a distorted image. We have recently developed a sequential Bayesian filtering approach to resolve this time difference problem [14].

Another known problem in the HS-AFM is the so-called “parachuting” artifact [15,16]. We illustrate it for a HS-AFM image of actin filaments obtained with relatively high scanning speed (Fig. 1A). The image taken from the forward-scanning (Fig. 1A left) exhibits horizontal scars extending from the actin filament to the right direction, which are designated as the parachuting artifact. In the backward-scanning image (Fig. 1A right), a clear parachuting artifact appears at the left side of the actin filament. When the probe scans the specimen slowly enough, the probe tip can go up and down alongside the specimen surface following the specimen surface accurately (Fig. 1B left). On the other hand, when the probe scanning velocity is higher and the specimen height suddenly decreases along the scanning line, the probe tip cannot fall sufficiently quickly so that some portion of the downhill surface is not sensed by the probe (Fig. 1B right). During this duration, the recorded height must be higher than that has to be, which is called the parachuting artifact. If one chooses a slower scanning velocity, one can avoid this artifact, but one also loses the time resolution. Therefore, to achieve high temporal resolution, we need to face the parachuting artifact problem and resolve it.

Figure 1 

The parachuting effect in HS-AFM image. A) A pair of HS-AFM images of actin filament from the forward- (left) and backward- (right) scans obtained in a single two-way scan. Representative areas of the parachuting artifact are marked by cyan dashed oval. Scanning area: 200×200 nm² with 100×100 pixels; frame time: 0.5 s (line rate: 200 Hz). B) Schematic cartoons of the parachuting artifact. Left: With a low-speed scan, the tip hits the surface object at every pixel point. Right: With a high-speed scan, the tip may miss hitting the object when the object height rapidly decreases along the scanning line.

Our main purpose here is to develop a computational method to detect and remove the parachuting artifact from HS-AFM data with two-way scanning. Because of the mechanism that caused the parachuting artifact, it should appear only when the specimen height decreases along the scanning direction. Thus, if a height at a pixel is hampered by the parachuting artifact in the forward-scanning, the same pixel would unlikely contain the parachuting artifact in the backward-scanning image (Fig. 1A). Then, our idea is to use two-way scanning data to resolve the parachuting artifact. When a pixel height from one-way scanning is detected as the parachuting artifact, we throw it away and use the height data from the reverse scanning. By this way, we expect to recover the missing height information due to the parachuting in one-way scanning. Similar stitching of two scanning data was previously investigated in the data processing for AFM images with a slantingly-mounted nanotube tip [17–19]. Notably, in the HS-AFM experiments so far, the backward-scanning data were not commonly utilized for analysis primarily due to the nonlinearity and the hysteresis effect of piezo; between the forward- and backward-scanning measurement, the same voltage value does not precisely correspond to the same position along the scanning line. In fact, when we compared the two-way scanning images, we observed that the piezo hysteresis hampers a straightforward merge of the two-way scanning data. To this end, we used a variant of previously-developed computational methods to correct the hysteresis and to align forward- and backward-scanning images only from the two-way scanning data of the target specimen [20–23].

In this paper, we begin with the development of the detection method of the parachuting artifact, using an actin filament HS-AFM image in Fig. 1A, as an example. We define the parachuting decision boundary lines beyond which the region is assigned as the parachuting area. Next, we analyze the piezo hysteresis curve and examined a method to infer the alignment of the two-way scanning data solely using the two-way scanning HS-AFM data. Then, we propose a method to make a processed HS-AFM image free from the parachuting artifact, using the two-way scanning images. Finally, we test our methods for some real HS-AFM images of actin filaments, chaperonin, and duplex DNA.

Materials and Methods

HS-AFM Measurements of Actin Filament

We used a laboratory built high-speed atomic force microscope (HS-AFM) as described previously [24]. In brief, a glass sample stage (diameter, 2 mm; height, 2 mm) with a thin mica disc (1.5 mm in diameter and 0.05 mm in thickness) glued to the top by epoxy was attached onto the top of a Z-scanner by a drop of nail polish.

For the observation of actin filaments, a freshly cleaved mica surface was treated for 5 min with 0.01% (3-aminopropyl) triethoxysilane (Shin-Etsu Chemical) diluted with milli-Q water. After rinsing the surface with drops of milli-Q water (20 μl×5), the solution was replaced with buffer A (25 mM KCl, 2 mM MgCl₂, 1 mM EGTA, 20 mM Imidazole–HCl, pH 7.6). A drop (2 μl) of actin filaments (ca. 1 μM), which were purified from rabbit skeletal muscle and stabilized with phalloidin [9], diluted with buffer A was deposited for 10 min. After rinsing the surface with buffer A of 20 μl, the sample stage was immersed in a liquid cell filled with buffer A of 60 μl. We used a single HS-AFM measurement for the results in Figures 1–4, S1–S3, Movie S1.

For the observation of GroEL, we prepared the surface specimen as described previously [25] with modifications. A drop (2 μl) of GroEL (Sigma-Aldrich) diluted to ca. 300 nM with buffer B (100 mM KCl, 5 mM MgCl₂, 10 mM DTT, 25 mM Tris-HCl, pH 7.5, 10% glycerol) was deposited on a freshly cleaved mica surface for 5 min. After rinsing the surface with buffer C (5 mM MgCl₂, 10 mM DTT, 25 mM Tris-HCl, pH 7.5, 10% glycerol) of 20 μl, the sample stage was immersed in a liquid cell filled with buffer C of 60 μl.

For the observation of duplex DNA, we prepared the surface specimen as described previously [26] with modifications. A drop (2 μl) of the 600-bp duplex DNA diluted to ca. 1 nM with milli-Q water was deposited for 3 min on a lipid bilayer containing DPPC, DPTAP and biotin-cap-DPPE at mixing ratio of 89:10:1 (weight ratio) formed on a mica surface. After rinsing the surface with buffer D (30 mM KCl, 5 mM MgCl₂, 10 mM Tris-HCl, pH 7.5) of 20 μl, the sample stage was immersed in the liquid cell filled with buffer D of 60 μl. After immersing the sample stage in the liquid cell, HS-AFM imaging was carried out in the tapping mode.

We used small cantilevers (BL-AC10DS-A2, Olympus, Tokyo) whose spring constant, resonant frequency in water, and quality factor in water were ~0.1 N/m, ~0.5 MHz, and ~1.5, respectively. The probe tip was grown on the original tip end of a cantilever through electron beam deposition using ferrocene and was further sharpened using a radio frequency plasma etcher (Tergeo, PIE Scientific LLC., USA) under an argon gas atmosphere (Direct mode, 10 sccm and 20 W for 1.5 min). The cantilever’s free oscillation amplitude A₀ and set-point amplitude A_s were set at ~2 nm and ~0.9×A₀, respectively.

Piezo Hysteresis Measurements

The hysteresis of the X-scanner was evaluated by measuring the displacement of the X-scanner, as previously described [27,28]. Briefly, a small mirror (~1.5×1.5×0.3 mm³), which is made from a cantilever chip, was glued with epoxy to the side of a glass AFM sample stage (diameter, 2 mm; height, 2 mm). The glass stage with the mirror was attached onto the top of a Z-scanner by a drop of nail polish so that the small mirror roughly faced to the direction of the X-scanner displacement. The X-scanner was moved by the same input signal as when observing the actin filaments (i.e., 200 nm at 210 Hz), and the displacement of the X-scanner was measured with a laser displacement meter (ST-3761, IWATSU, Japan). The input and displacement signals were simultaneously recorded with a digital oscilloscope (DS-5354, IWATSU, Japan). Note that the measured displacement was slightly smaller than the actual displacement as the mirror cannot be aligned to perfectly match the direction of the X-scanner displacement.

Results and Discussions

Detection of the Parachuting Artifact

First, we develop a method to detect the parachuting artifact solely from given HS-AFM image data. As already described, the parachuting artifact appears when the surface specimen height decreases along the scanning line more rapidly than the probe tip can follow. In this duration, the probe does not touch the specimen and is expected to decrease its height at its maximum speed, which should be a constant depending on the scanning velocity and the instrument performance. Specifically, if the scanning velocity along the x-axis is v [pixel/μs] and the maximum velocity of the probe to fall along z is u [Å/μs], the height data due to the parachuting can be approximated as a line with a slope of –u/v [Å/pixel], thus z [Å]=–(u/v)×[pixel]+constant. Possibly, this slope value can be measured, but the measurement is not always easy. For example, if you obtain HS-AFM videos previously measured by others, you may not get the necessary information. In this study, we seek a computational method to infer the slope solely from the acquired AFM image data, so that anyone can easily apply the method after the measurement.

We assume that a HS-AFM image is acquired in a forward-directed raster scan. For each line (along x-axis), we calculate the difference in the height ∆z=z(j+Δ)–z(j) from the j-th pixel to the j+Δ-th pixel along the i-th scanning line for Δ≥0. We repeat this for all the lines in the image. Then, averaging over i and j, we can obtain the histogram H(∆,∆z) in the two dimension Δ and ∆z. (exemplified in Fig. 2A for the same HS-AFM data as Fig. 1A). In the heatmap, from the origin (∆z=0, ∆=0), we recognize a border line in the negative ∆z side (bluish). Slightly above the border, we found a ridge line (reddish) suggesting that the probe tip tends to fall down at this speed with a high probability. Since this common rate is unlikely due to the shape of the specimen, we regard it as the signature of the parachuting artifact. When we average the heatmap for 400 images from the HS-AFM video of actin filament, we obtain Fig. 2B, which shows the characteristic border similar to the case of the single image, although the averaging makes the ridge somewhat unclear. Based on visual inspection of these heatmaps, we define the three “decision boundary” lines, ∆z=–0.5∆, ∆z=–(2/3)∆ and ∆z=–(2/3)∆–5/6, (blue, orange, and green lines, respectively, in Fig. 2AB). The first (third) line correspond to the most generous (stringent) boundaries for the detection of parachuting artifact; we will compare these three decision lines.

Figure 2 

Detection of the parachuting area from the HS-AFM image. A, B) Histograms H(∆,∆z) of the image height change ∆z along the scanned pixel for the forward-scanning data of the same HS-AFM image of actin filament as Fig. 1. The horizontal axis represents the change ∆ in pixel along x axis (parallel to the scanning line). The vertical axis is the image height change ∆z=z(x+∆,y)–z(x,y) (Å). The color represents the number of occurrences of (∆,∆z) in the 305-th frame image (A) and in all the 400 images (B). Straight lines are three decision lines of the parachuting artifact; blue ∆z=–0.5∆ orange ∆z=–(2/3)∆, and green ∆z=–(2/3)∆–5/6. (C) Area detected as the parachuting artifact are drawn in white for the three decision lines (the right three images), together with the original HS-AFM image (the left most).

For an image, we define the parachuting area as the set of all the points that are below the decision boundary line. For the case of actin filament, we depict the parachuting area in Fig. 2C for the three decision boundary lines. The parachuting area that can be inferred by our eye is extracted correctly.

Similarly, we found the characteristic border line in the heatmap obtained from the backward-directed raster-scanning data. With the same definition of the decision boundaries, we can extract the parachuting area from the same HS-AFM measurement as Fig. 2 (Fig. S1). Of note, we focus on the positive slope in the heatmap in the case of backward-directed scanning image.

Inference of the Hysteresis Relationship Between Forward- and Backward-scanning Images

Once we identify the parachuting artifact area in the forward- and backward-scanning HS-AFM data, next we seek to eliminate the artifact and to obtain the processed image. Since the parachuting artifact appears only when the specimen height decreases quickly along the scanning line, we assume that it appears at most in one of the two-way scans. In addition, we expect that, when the parachuting artifact appears scanning in one-direction, the probe height recorded in that direction scanning is higher than that in the opposite direction scanning. To this end, a quick and easy treatment is simply to take the lower height of the two-way scanning heights for each pixel. This simple treatment resulted in Fig. 3A, where we could eliminate the parachuting artifact, apparently. However, we noticed that the top left part of the actin filament appears thinner than the other region. Comparing the cyan-marked part with the magenta-marked part in Fig. 3A, we can easily notice the thickness of the actin filament is markedly different. We anticipated that this another well-known type of artifact arises due to non-linear hysteresis effect of the piezoelectric element (piezo). The piezo used in the scanning extends its length as the voltage is applied. The relationship between the piezo distance and the applied voltage is approximately, but not exactly, linear. Moreover, the distance-voltage relationship slightly differs between the extending and the shrinking phases, which is called the hysteresis. Namely, with the same voltage applied, the x-coordinate of the probe tip slightly differs between the forward- and backward-scanning phases. The hysteresis in the piezo and its effect on the AFM image have been anticipated previously.

Figure 3 

Inference of the hysteresis relationship between forward- and backward-scanning images. A) The image produced by taking the minimum of the forward- and backward-scanning heights for 305-th frame images of actin filament HS-AFM data, with no treatment of hysteresis. In the processed image, the filament apparently looks thinner at the region marked by cyan triangles compared to that by magenta triangles. B) A hysteresis curve directly measured. The horizontal and vertical axes are the input signal in the X-scanner in voltage and the displacement signal measured by laser displacement meter (the measured voltage can be converted to the length by a factor 100 nm/V). C, D) Schematics to infer the hysteresis. E) Comparison of two score functions and three weightings in inferring hysteresis from the HS-AFM data of actin filament. The optimal hysteresis relationship between the forward-scanning x-axis (the horizontal axis) and the backward-scanning (the vertical axis) with use of two score functions (RMSD in the first raw and the cosine similarity for the second raw) and different weightings (f_i=z_i at the left column, f_i=z_i² at the center, and f_i=exp(z_i) at the right). Results from the frames 1st–800th (the entire video), 105th–505th, 205th–405th, 255th–355th, and 295th–315th, are in red, green, blue, magenta, and cyan, respectively.

We set up the system to directly measure the nonlinear hysteresis of the X-scanner. As expected, the X-scanner displaces differently in the forward- and backward-scanning phases (Fig. 3B). While the direct measurement of the piezo hysteresis is possible, the hysteresis depends on the instrument as well as the operation parameters so that the hysteresis must be measured nearly every time prior to the product HS-AFM measurement, which is practically troublesome. It would be much more convenient if one can infer the hysteresis solely from the acquired forward- and backward-scanning image data. Such methods have been developed previously for AFM image data [20–23]. Here, we employed a variant of such a method [20,21] adapted to HS-AFM video data.

The piezo hysteresis results in mismatches in specimen heights recorded at the j-th column between forward- and backward-scanning data. In other words, the height of one part of specimen is recorded in slightly different columns in the two image-data (Fig. 3C). The measured hysteresis curves in Fig. 3B suggests that the mismatch tends to increase in the middle of scanning and disappear at both end columns. We assume that the hysteresis is identical in all the rows in the image. Then, for the j-th column vector in the forward-scanning image data, we seek the closest column vector (j’), in the backward-scanning image data (Fig. 3D). Thus, j’ is a function of j that matches, which is termed as the hysteresis relationship j’(j) hereafter.

As a measure to quantify the closeness between column vectors of two-way scanning data, f and b, we examine two similarity measures; the root-mean-square-deviation (RMSD) and the cosine similarity, where f_i and b_i are the i-th element (row) of the column vectors of the forward- and backward scans. Note, however, that f_i (and b_i) are not necessarily identical to the measured height, but can be any monotonically increasing function of the height, as we explore in the next paragraph. The RMSD is the root of the sum of the squared difference between the values in the forward- and backward-scanning column vectors. Thus, the more similar the two column vectors are, the smaller the RMSD is. The cosine similarity is the scalar product of the normalized column vectors,

  

$$\cos \left(\mathbf{f},\mathbf{b}\right)=\frac{{\sum }_i^{}{f}_i b_i}{\sqrt{{\sum }_i^{}{f}_i^2} \sqrt{{\sum }_i^{}{b}_i^2}}$$(1)

The cosine similarity increases as the two vectors become close. In the comparison, the pixels that contain parachuting artifact in one direction were not excluded in the estimate.

We examined three different mappings from the height vectors z to the column vector f (and b in the case of the backward-scan) to be used in the above formula. Obviously, the simplest choice is f_i=z_i. However, we anticipated that this choice may be sensitive to the noise, which is not small in the HS-AFM images, generally. We note that, in the AFM images, specimen information is contained mostly at pixels with larger heights, whereas the pixels with smaller heights can be dominated by noise. Thus, to emphasize the former, we propose the two non-linear mappings, f_i=z_i², and f_i=exp(αz_i) with α=1/Å.

Additionally, we examined the number of frames need to be used to infer the hysteresis relationship. Generally, the signal-to-noise ratio in the inference should be higher if we use more frames. However, in real-world AFM measurement, the operator tends to change measurement parameters during the measurement so that the hysteresis curve may change intermittently. Thus, the use of smaller number of frames are advantageous to infer the weakly time-dependent hysteresis relationship. To this end, we seek a method that can robustly infer the hysteresis relationship with fewer number of frames. In the test shown below, we calculated the hysteresis relationship for the HS-AFM video of actin filament. This video contains 800 forward- and backward-scanning frames, in which we focused the 305-th images for the removal of parachuting artifact (Fig. 1A). Centered at this time point, we examined four numbers of frames to be used; the 21 images (the 295th to 315th images), the 101 images (the 255th to 355th images), the 201 frames (the 205th to 405th images), and the 401 frames (the 105th to 505th images), in addition to the whole images (the 1st to 800th images). We sought the best alignment j’(j) in the range that |j’(j)-j|≤10.

We performed a numerical test for the HS-AFM video of actin filament with the two distance measures, three f←z mappings, and five numbers of frames used, of which results are shown in Fig. 3E and Fig. S2. We first describe the case that used the RMSD as a measure. With the simple f_i=z_i mapping (Fig. 3E top left), the inferred mismatch gap |j’(j)-j| at the central column exceeded 5 pixels which is the experimentally measured maximum mismatch gap in Fig. 3B, for all the five numbers of frames used. We checked that this setup resulted in the merged image in which the actin filament is clearly too thick (Fig. S2). The use of a non-linear mapping f_i=z_i² slightly improved the result (Fig. 3E top center), where the hysteresis shows a reasonable smooth curve with a maximal mismatch gap of 5 pixels when we used the entire 800 frames, but, the gap exceeded 5 pixels for the other cases of fewer frames. When we used the exponential mapping, f_i=exp(z_i), the RMSD value sometime exceeded our threshold that could cause the numerical problem, which precluded us from getting the optimal alignment. Then, the hysteresis curve becomes wobbly (Fig. 3E top right).

When we tested the cosine similarity as a measure, we found the results tend to be more robust with respect to the number of frames used. With use of f_i=z_i and f_i=z_i², however, the inferred gap at the central column exceeded 5 pixels, thus overestimating the hysteresis. Contrary to these, when we used f_i=exp(z_i), we found the maximal mismatch gap becomes 5 pixels and the hysteresis curve is smooth and is robust against the number of frames used (Fig. 3E bottom right). This is probably because the exponential function enlarges the weight of the pixels at which actin filament exists, relative to pixels with only noise. Notably, the use of the exponential function makes the similarity measure invariant to the absolute height; a shift in the z-value, z₀ leads to the multiplying factor exp(z₀) in the column vectors, which disappears upon the normalization in the formula of the cosine similarity. This invariance could help the performance of this setup since the absolute height of the stage is not accurately known. In summary, we conclude that the best method among tested is to use the cosine similarity between the exponentially scaled pixel height column data. Hereafter, we solely used this method.

Removal of the Parachuting Artifact

With the methods for detection of the parachuting area and for the inference of the hysteresis relationship between the two-way scanning data in hand, we are ready to remove the parachuting artifact using the two-way scanning HS-AFM data, f_ij and b_ij. Here, we examine methods with the same actin filament images as above (Fig. 1A).

First, using the three parachuting decision boundaries from the generous one to the stringent one (Fig. 2C), we detected the parachuting areas (Fig. 4A top line).

Figure 4 

Removal of the parachuting artifact from the HS-AFM image of actin filament taking into accounts the hysteresis relation. A) (top) Detection of parachuting area in both forward- and backward-scanning data with three decision lines taking into accounts the hysteresis relation for the 305th frame. The hysteresis relation was inferred using 1st–800th frames with the cosine similarity of exp(z). The white and black regions are detected as the parachuting area in the forward- and backward-scanning data. (bottom) The processed images by merging the two-way scan data using the soft-min function. The right-most panel is obtained simply by using the soft-min. β=2.0 was used. B) The same image as the bottom line of A) with a different color scheme; only the positive height region is colored. C) A cross section of the images in B) along the actin filament (dashed green line in the top left image of B). The red, green, blue, and black curves are from the parachuting decision lines, ∆z=–0.5∆, ∆z=–(2/3)∆, ∆z=–(2/3)∆–5/6, and no use of the parachuting removal treatment, respectively. The double-arrows just indicate the difference between the results with and without the removal of parachuting artifact. D) The processed images in different frames: From the top the 113rd, 123rd, 197th, and 305th frames. β=2.0, and ∆z=–(2/3)∆, were used.

The optimal hysteresis relationship between the forward- and backward-scanning data is obtained by the maximal cosine similarity of the exponential scale of the pixel height. To reduce the necessary number of frames to infer the hysteresis curve, we approximate the hysteresis relationship by a quadratic function and restraint the two ends having no mismatch gap; we used the standard least square fitting to the obtained hysteresis relationship. With this approximation by a quadratic function, we can robustly obtain the hysteresis relationship even with one pair of forward- and backward-scanning data.

Using the hysteresis relationship, for the j-th column of the forward-scanning column, we align the backward-scanning column j’ that gives the highest similarity. For each pixel (i,j), if the data at (i,j) in the forward-scanning is assigned as the parachuting artifact, we do not use this artifact data and simply use the data from the backward-scan and thus the processed image is I_ij=b_ij’. Otherwise, if the data at (i,j’) in the backward-scanning data is assigned as the artifact, we set I_ij=f_ij. If neither (i,j) in the forward-scanning nor (i,j’) in the backward-scanning data is assigned as the parachuting artifact, we average the two-way scanning data to obtain the processed image I_ij. For the averaging, we use a modified form of the soft-minimum function as

  

$$I_{ij}=\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{n}(f_{ij},b_{ij\text{'}})$$(2)

where the modified softmin function is defined as

  

$$\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{t}\mathrm{m}\mathrm{i}\mathrm{n}(\mathrm{x},\mathrm{y})=-{\beta }^{-1}\mathrm{l}\mathrm{n}(e^{-\beta x}+e^{-\beta y})+{\beta }^{-1}\left(\mathrm{l}\mathrm{n} 2 \right)e^{-{\beta }^2{(x-y)}^2}$$(3)

where the last term is derived as the correction when x and y are close each other (namely, when x is equal to y, the softmin should be identical to x). This takes an average of x and y with a larger weight for the smaller of x and y. The parameter β>0 controls the weight; In the limit of β→0, this softmin converges to the arithmetic average, whereas it becomes the normal minimum function in the limit β→∞. We note that, even with the correction term, the softmin function takes a slightly smaller value of the minimum of x and y when β(x–y) is comparable to unity. Here, we test the appropriate value of β, together with the parachuting decision boundaries.

The resulting images are shown in Fig. 4A and Fig. S3. We found that with a broad range of β values and any of the three parachuting decision boundaries, the parachuting artifact is apparently removed and the processed images look similarly good (Fig. S3). Fortunately, the results are robust for different choices of the decision boundaries. In fact, we realized that the softmin function itself can diminish the parachuting artifact markedly as is shown in the right-most column. The larger value of β tends to make the image sharper although it is not very obvious from the images in the color scheme in Fig. 4A, Fig. S3. (Note that we processed the height data, but not the color images. The former is not affected by the color scheme at all)

To compare these images more closely, we replot the images with β=2Å^–1 with the three decision boundaries as well as the result just from the softmin function of the two-way scans using a different color scheme; the new color scheme focuses large-height regions (about one third of the entire height range) (Fig. 4B). Near the top left region of the replotted images, we would notice that the actin filament height is slightly smaller than other parts in the image when the parachuting treatment was not applied (the bottom right image). To see it more clearly, we plot the surface height of a cross-section along the filament, employing an interpolation function in the two-dimension;

  

$$\begin{array}{c}{Z}_{x, y}=\left(1-q\right)\left\{\left(1-p\right)z_{int\left(x\right), int\left(y\right)}+p{z}_{int\left(x\right)+1, int\left(y\right)}\right\}+ q\{\left(1-p\right)z_{int\left(x\right), int\left(y\right)+1}+p{z}_{int\left(x\right)+1, int\left(y\right)+1}\}\hfill \\ \hfill (p=x-int\left(x\right), q=y-int\left(y\right))\end{array}$$(4)

The results in Fig. 4C shows that, indeed, without the use of parachuting treatment, the height of the actin filament was slightly lower.

We applied the same data processing to all the other frames in the same actin filament HS-AFM video (Fig. 4D for some representative frames and S1 Movie for the processed video). Here, we used β=2Å^–1 in the softmin function and the parachuting decision line ∆z=–(2/3)∆. We note that, in addition to the removal of the parachuting artifact, the treatment also reduced the noise by averaging the two-way scanning data. Interestingly, we observed the sudden breakage of the actin filament from the intact filament in the 113rd frame to the broken filament in the 123rd frame. The 197th frame also shows local disassembly of actin oligomers, perhaps due to the mechanical interaction by the probe tip.

Applications to Other HS-AFM Images

So far, we have solely used the actin filament HS-AFM data for the development and the validation of the method. Now, we applied the same procedure to other cases.

First, we obtained the two-way scan HS-AFM data for bacterial chaperonin, GroEL. The GroEL is known to form a homo 14-mer complex, in which two ring-shaped heptamers stack in a head-to-head configuration making a cylinder shape (Fig. 5A). There is a large cavity along the cylinder axis, which is used to capture its substrate proteins. We measured GroEL complexes with the two-way scan HS-AFM using relatively fast scan speed. The HS-AFM video contains 0th–63th frames of two-way scanned AFM images with 150x150 pixels (a representative pair of images at the 50th frame of the video in Fig. 5B). In Fig. 5B, ring-shaped molecules in the upper half as well as in the right region (bright orange) are supposed to be head-to-head docked full GroEL complexes of ~14 monomers, whereas some ring-shaped molecules with much darker orange near the bottom of the field may correspond to one-layered GroEL ring with ~7 monomers. The full GroEL complexes look circular with little fine structural feature. However, only for limited samples, we barely recognize the central cavity. More importantly in the current study, we see clear parachuting artifact as horizontal scars. We applied the same procedure as above. Using the heatmap plot of the height change, we obtained the parachuting decision lines (Fig. 5C and Fig. S4A for the forward- and backward-scanning data, respectively). We then obtained the hysteresis relationship between the two-way scan data using the cosine similarity of the exponentially scaled pixel height, which is fitted by the quadratic function (Fig. 5D). Based on the parachuting decision lines and the hysteresis relationship, we plot the detected parachuting area as well as the processed image (Fig. 5E for the case of ∆z=–(2/3)∆–5/6, and Fig. S4B for the other two decision lines). We found that much of the scars, hallmarks of the parachuting artifact, were removed in the processed image. In addition, we notice some image improvement; boundaries between different GroEL complexes are clearer than the raw images. (Fig. 5F). This may be related to the improvement of the detection of concave shapes. In the raw data, the concave surface is likely to be affected by the parachuting artifact, and thus the removal of the parachuting artifact can enhance the potential to detect the concave shape. In addition, averaging over two images simply increases the S/N ratio, which also contributed to increases the recognition.

Figure 5 

The image processing results for a HS-AFM video of GroEL. A) The X-ray crystal structure of GroEL 14-mer complex from the top and from the side (PDB ID: 1SS8). B) A forward- and backward-scanning HS-AFM images for the 50th frame of the video. Scanning area: 200×200 nm² with 150×150 pixels; frame time: 0.66 s (line rate: 225 Hz). The red squared area is expanded in F). C) The histogram to assess the parachuting artifact for the forward-scanning image of the 50th frame. The blue, orange, and green lines are the decision lines of the parachuting artifact; blue ∆z=–0.5∆ orange ∆z=–(2/3)∆, and green ∆z=–(2/3)∆–5/6. D) The hysteresis relation obtained from the entire video (the 0th to 63rd frames) using the cosine similarity score with the exponential weight (red), and its quadratic fitting curve (green). E) The parachuting artifact area detected with ∆z=–(2/3)∆–5/6 and the hysteresis curve in D), together with the image processed to remove the artifact. F) A closeup view of the red-squared area in B). The left and middle panels are the forward- and backward-scanning images, respectively. The right panel represents the processed image that reveals ring-shaped 7 subunits, indicated by black arrows.

Finally, we obtained the two-way scan HS-AFM data for duplex DNA (Fig. 6). Since the height of duplex DNA bound on the surface is much lower than those of actin filament and the GroEL complexes, we do not see clear pattern of the parachuting artifact. While our method did detect the parachuting area next to the duplex DNA, the processed image is not noticeably different from the raw images. Thus, for specimen of which height is not markedly large, the parachuting artifact would not affect the HS-AFM images, and thus the removal of it is not essential.

Figure 6 

The detection and the removal of parachuting artifact in the HS-AFM data of duplex DNA. A) A representative pair of forward-scanning (top) and backward-scanning (bottom) image data for the duplex DNA. Scanning area: 150×150 nm² with 80×80 pixels; frame time: 0.5 s (line rate: 160 Hz). B) Histogram H(∆,∆z) of the image height change ∆z along the scanned pixel for the forward-scanning (left) and backward-scanning (right) data of HS-AFM. C) The mapping between the forward-scanning (the horizontal axis) and backward-scanning (the vertical axis) x-axes by the measure of the cosine similarity and the exponential scaling. The direct mapping in red, and the fitted data in green. D) The detected parachuting area (top) and the processed images with three parachuting decision lines, and without the parachuting treatment but using the soft-min function (bottom).

Conclusion

To capture fast motions of biomolecules, we desire to achieve as high temporal resolution as possible in the HS-AFM measurement. When the probe tip scans the stage at high speed, we often encounter the so-called parachuting artifact. We developed a computational method to detect and remove the parachuting artifact in HS-AFM images using the two-way scanning data. To merge the two-way scanning images, we found it important to take into accounts the piezo hysteresis effect; we employed a method to align the forward- and backward-scanning images. We then examined our method for HS-AFM videos of actin filaments, GroEL complexes, and duplex DNAs. Our method can remove the parachuting artifact from the measured two-way scan HS-AFM video data making the processed video free from the parachuting artifact. The method is general and fast so that it can easily be applied to any HS-AFM videos with two-way scanning data. A GUI-based image processing tool for HS-AFM video data that includes the current study will be made available.

Finally, we note that the proposed method has limitations as well. First, it has been suggested that the backward-scanning image tends to have lower quality than the forward-scanning image, due to the inherent architecture of the apparatus [29]. Thus, using the backward-scanning data can distort the processed image in some cases. A recent study proposed to skip the backward-scanning process to speed up the sampling rate and reduce the damage to the specimen [29]. Second, specimen molecules can move during the two-way scanning, which can distort the image. In principle, we can combine the current method with a recently developed Bayesian approach to take into accounts of data asynchronicity [14].

Conflict of Interest

The authors declare no competing interests.

Author Contributions

S.K. and S.T. designed the work. S.K. created the new software. K.U. and N.K. measured the HS-AFM and piezo hysteresis curve. S.K. performed the calculations and analyzed the data. S.K, N.K., and S.T. interpreted the data. All the authors wrote and checked the manuscript.

Data Availability

The evidence data generated during the current study are available from the corresponding author on reasonable request.

Acknowledgements

This work was supported by the Japan Science and Technology Agency (JST) grant (JPMJCR1762), KAKENHI, Japan Society for the Promotion of Science [Grant Number 21J00021 and 22K15070 (to SK), 20H00327 (to N.K.), and 20H05934 and 21H02441 (to S.T.)].

References

• [1]   Shi,  Y. A glimpse of structural biology through X-ray crystallography. Cell 159, 995–1014 (2014). https://doi.org/10.1016/j.cell.2014.10.051
• [2]   Bai,  X.-C.,  McMullan,  G.,  Scheres,  S. H. W. How cryo-EM is revolutionizing structural biology. Trends Biochem. Sci. 40, 49–57 (2015). https://doi.org/10.1016/j.tibs.2014.10.005
• [3]   Moerner,  W. E.,  Fromm,  D. P. Methods of single-molecule fluorescence spectroscopy and microscopy. Rev. Sci. Instrum. 74, 3597–3619 (2003). https://doi.org/10.1063/1.1589587
• [4]   Mazal,  H.,  Haran,  G. Single-molecule FRET methods to study the dynamics of proteins at work. Curr. Opin. Biomed. Eng. 12, 8–17 (2019). https://doi.org/10.1016/j.cobme.2019.08.007
• [5]   Bacic,  L.,  Sabantsev,  A.,  Deindl,  S. Recent advances in single-molecule fluorescence microscopy render structural biology dynamic. Curr. Opin. Struct. Biol. 65, 61–68 (2020). https://doi.org/10.1016/j.sbi.2020.05.006
• [6]   Ando,  T.,  Uchihashi,  T.,  Kodera,  N. High-speed AFM and applications to biomolecular systems. Annu. Rev. Biophys. 42, 393–414 (2013). https://doi.org/10.1146/annurev-biophys-083012-130324
• [7]   Ando,  T. High-speed atomic force microscopy and its future prospects. Biophys. Rev. 10, 285–292 (2018). https://doi.org/10.1007/s12551-017-0356-5
• [8]  Shimizu, M., Okamoto, C., Umeda, K., Watanabe, S., Ando, T., Kodera, N. An ultrafast piezoelectric Z-scanner with a resonance frequency above 1.1 MHz for high-speed atomic force microscopy. Rev. Sci. Instrum. 93, 013701. (2022). https://doi.org/10.1063/5.0072722
• [9]   Kodera,  N.,  Yamamoto,  D.,  Ishikawa,  R.,  Ando,  T. Video imaging of walking myosin v by high-speed atomic force microscopy. Nature 468, 72–76 (2010). https://doi.org/10.1038/nature09450
• [10]   Ando,  T.,  Uchihashi,  T.,  Kodera,  N.,  Yamamoto,  D.,  Miyagi,  A.,  Taniguchi,  M., et al. High-speed AFM and nano-visualization of biomolecular processes. Pflugers Arch. 456, 211–225 (2008). https://doi.org/10.1007/s00424-007-0406-0
• [11]   Shibata,  M.,  Yamashita,  H.,  Uchihashi,  T.,  Kandori,  H.,  Ando,  T. High-speed atomic force microscopy shows dynamic molecular processes in photoactivated bacteriorhodopsin. Nat. Nanotechnol. 5, 208–212 (2010). https://doi.org/10.1038/nnano.2010.7
• [12]   Uchihashi,  T.,  Iino,  R.,  Ando,  T.,  Noji,  H. High-speed atomic force microscopy reveals rotary catalysis of rotorless F1-ATPase. Science 333, 755–758 (2011). https://doi.org/10.1126/science.1205510
• [13]   Miyagi,  A.,  Chipot,  C.,  Rangl,  M.,  Scheuring,  S. High-speed atomic force microscopy shows that annexin V stabilizes membranes on the second timescale. Nat. Nanotechnol. 11, 783–790 (2016). https://doi.org/10.1038/nnano.2016.89
• [14]   Kubo,  S.,  Kato,  S.,  Nakamura,  K.,  Kodera,  N.,  Takada,  S. Resolving the data asynchronicity in high-speed atomic force microscopy measurement via the Kalman Smoother. Sci. Rep. 10, 1–16 (2020). https://doi.org/10.1038/s41598-020-75463-1
• [15]   Kodera,  N.,  Sakashita,  M.,  Ando,  T. Dynamic proportional-integral-differential controller for high-speed atomic force microscopy. Rev. Sci. Instrum. 77, 083704 (2006). https://doi.org/10.1063/1.2336113
• [16]   Ando,  T.,  Uchihashi,  T.,  Fukuma,  T. High-speed atomic force microscopy for nano-visualization of dynamic biomolecular processes. Prog. Surf. Sci. 83, 337–437 (2008). https://doi.org/10.1016/j.progsurf.2008.09.001
• [17]   Fu,  J.,  Dixson,  R.,  Orji,  G.,  Vorburger,  T.,  Nguyen,  C. V. Linewidth measurement from a stitched AFM image. AIP Conference Proceedings 788, 421–426 (2005). https://doi.org/10.1063/1.2799408
• [18]   Chu,  W.,  Fu,  J.,  Dixson,  R.,  Vorburger,  T. Linewidth measurement based on automatically matched and stitched AFM images. AIP Conference Proceedings 931, 407–412 (2007). https://doi.org/10.1063/1.2799408
• [19]   Cho,  S. J.,  Ahn,  B. W.,  Kim,  J.,  Lee,  J. M.,  Hua,  Y.,  Yoo,  Y. K., et al. Three-dimensional imaging of undercut and sidewall structures by atomic force microscopy. Rev. Sci. Instrum. 82, 1–6 (2011). https://doi.org/10.1063/1.3553199
• [20]   Chu,  W.,  Fu,  J.,  Dixson,  R.,  Orji,  G.,  Vorburger,  T. A moving window correlation method to reduce the distortion of scanning probe microscope images. Rev. Sci. Instrum. 80, 1–5 (2009). https://doi.org/10.1063/1.3189041
• [21]   Fu,  J.,  Chu,  W.,  Dixson,  R.,  Orji,  G.,  Vorburger,  T. Correction of hysteresis in SPM images by a moving window correlation method. AIP Conference Proceedings 1173, 280–284 (2009). https://doi.org/10.1063/1.3251234
• [22]   Yothers,  M. P.,  Browder,  A. E.,  Bumm,  L. A. Real-space post-processing correction of thermal drift and piezoelectric actuator nonlinearities in scanning tunneling microscope images. Rev. Sci. Instrum. 88, 013708 (2017). https://doi.org/10.1063/1.4974271
• [23]   Zhang,  L.,  Chen,  X.,  Huang,  J.,  Li,  H.,  Chen,  L.,  Huang,  Q. A method to correct hysteresis of scanning probe microscope images based on a sinusoidal model. Rev. Sci. Instrum. 90, 023704 (2019). https://doi.org/10.1063/1.5052281
• [24]   Uchihashi,  T.,  Kodera,  N.,  Ando,  T. Guide to video recording of structure dynamics and dynamic processes of proteins by high-speed atomic force microscopy. Nat. Protoc. 7, 1193–1206 (2012). https://doi.org/10.1038/nprot.2012.047
• [25]   Yokokawa,  M.,  Wada,  C.,  Ando,  T.,  Sakai,  N.,  Yagi,  A.,  Yoshimura,  S. H., et al. Fast-scanning atomic force microscopy reveals the ATP/ADP-dependent conformational changes of GroEL. EMBO J. 25, 4567–4576 (2006). https://doi.org/10.1038/sj.emboj.7601326
• [26]   Yoshimi,  K.,  Takeshita,  K.,  Kodera,  N.,  Shibumura,  S.,  Yamauchi,  Y.,  Omatsu,  M., et al. Dynamic mechanisms of CRISPR interference by Escherichia coli CRISPR-Cas3. Nat. Commun. 13, 4917 (2022). https://doi.org/10.1038/s41467-022-32618-0
• [27]   Watanabe,  H.,  Uchihashi,  T.,  Kobashi,  T.,  Shibata,  M.,  Nishiyama,  J.,  Yasuda,  R., et al. Wide-area scanner for high-speed atomic force microscopy. Rev. Sci. Instrum. 84, 053702 (2013). https://doi.org/10.1063/1.4803449
• [28]   Marchesi,  A.,  Umeda,  K.,  Komekawa,  T.,  Matsubara,  T.,  Flechsig,  H.,  Ando,  T., et al. An ultra-wide scanner for large-area high-speed atomic force microscopy with megapixel resolution. Sci. Rep. 11, 1–11 (2021). https://doi.org/10.1038/s41598-021-92365-y
• [29]   Fukuda,  S.,  Ando,  T. Faster high-speed atomic force microscopy for imaging of biomolecular processes. Rev. Sci. Instrum. 92, 033705 (2021). https://doi.org/10.1063/5.0032948