Online ISSN : 2185-4610
Print ISSN : 0016-2590
ISSN-L : 0016-2590
Original Article
Validation of MALDI-TOF MS devices in reanalysis of unidentified pathogenic bacteria detected in blood cultures
Minako ImaiYukio KimuraDaiki TannoKyoichi SaitoMutsuko HondaYukiko TakanoKazutaka OhashiMasahiro ToyokawaNoboru OhanaYukio YamaderaHiroki Shimura
Author information

2020 Volume 66 Issue 2 Pages 103-112


In hospital microbial laboratories, morphological and biochemical analyses are performed to identify pathogenic microbes;however, these procedures lack rapidity and accuracy. Recently, Matrix-Assisted Laser Desorption/Ionization Time-of-Flight Mass Spectrometry (MALDI-TOF MS) has been clinically utilized, and is expected to enable rapid and accurate microbial identification. We aimed to validate two MALDI-TOF MS devices available in Japan: the VITEK-MS (BioMérieux) and the Microflex LT (Bruker Daltonics). Clinically isolated bacteria, 100 samples in all, detected in blood cultures but incompletely identified by conventional procedures, were reanalyzed using the two devices. The VITEK-MS and Microflex LT, respectively, identified 49% (49/100) and 80% (80/100) of the tested bacteria at the species level, as well as 96% (96/100) and 95% (95/100) at the genus level. Among those reidentified strains, 26% (26/100) at the species level and 88% (88/100) at the genus level were concordant with each other, though three strains were unmatched. Moreover, four bacterial strains were unable to be identified using the VITEK-MS, versus five using the Microflex LT. MALDI-TOF MS devices can provide more rapid and accurate bacterial identification than ever before;however, the characteristics of each system were slightly different;therefore, it is necessary to understand the difference in performance of MALDI-TOF MS models.


Blood culture is among the most important procedures in clinical microbiology for identifying pathogens that cause bloodstream infections (BSIs). Rapid and accurate identification of microbes is beneficial for patients, in that it leads to quicker determination of appropriate treatment for BSIs1,2). In clinical laboratories, morphological and biochemical analyses are typically performed to identify such pathogenic microbes, which may be distinguished by Gram stain, biochemical properties, and colony features such as color, shape, and smell. Even in the hands of experienced laboratory technicians, these protocols may lack rapidity and accuracy.

In recent years, Matrix-Assisted Laser Desorption/Ionization Time-of-Flight Mass Spectrometry (MALDI-TOF MS) has been introduced and utilized to identify pathogenic microbes isolated in clinical tests such as blood cultures3-5). MALDI-TOF MS applies a laser-based ionization technique in which large organic molecules are turned into ions, and their flight times are analyzed using a mass spectrometer6). The flight time of each ion differs according to its mass-to-charge ratio;therefore, the material composition of tested organic material, including microbes, can be inferred. Reference spectra aggregated from already-known microbes are aggregated and updated for homological analysis of unidentified specimens. When the spectral pattern of a sample matches one stored in the database, the tested microbe is identified.

In Japan, two types of MALDI-TOF MS devices, the VITEK-MS (BioMérieux, Marcy l’Étoile, France) and the Microflex LT (Bruker Daltonics, Bremen, Germany), have been approved for use in clinical hospital laboratories. Utilizing these two models, we reanalyzed clinically isolated pathogenic bacteria, 100 sample in all, which had been detected in blood cultures, but whose identification failed through conventional protocols. In the current study, we aimed to validate the MALDI-TOF MS devices against conventional protocols, as well as compare the performance of the two models available in Japan.

Materials and methods


Sample bacterial strains were collected via routine blood cultures, which were ordered by clinicians and performed in the clinical microbial laboratory at Fukushima Medical University Hospital from January 2013 to December 2015. A total of 106 bacterial strains were nominated for this study, which had not been fully identified by conventional procedures. The conventional protocols performed in our laboratory include Gram-staining, morphologic assessment of colonies, and comparison of their biochemical properties obtained from a MicroScan WalkAway System (Beckman Coulter, CA, USA), RAPID ID 32 STREP (BioMérieux, Marcy l’Étoile, France), and RapID ANA II System (Thermo Fisher Scientific, KS, USA). These incompletely identified bacteria were reported to clinicians based on bacterial genus or phenotypic features;e.g. Staphylococcus sp., Gram-positive chain cocci, Gram-negative glucose-fermenting rods, etc.


I) Preservation and re-culturing

The nominated sample bacterial strains were preserved at −80℃ using Microbank (Pro-Lab Diagnostics, Richmond Hill, Canada), a cryo-preservative vial system for the storage and retrieval of isolated bacterial strains. The strains were re-cultured on Trypticase Soy Agar with 5% Sheep Blood (Becton Dickinson Japan, Tokyo, Japan). Incubation procedures varied according to the characteristics of each sample strain. The general or noncommittal bacterial strains were aerobically incubated at 35℃ for 18 to 24 hours. The anaerobic bacterial strains were incubated at 35℃ using a Thermo Forma Anaerobic System (Thermo Forma, Marietta, OH, USA). The strains characterized as slow-growing were continuously cultivated at 35℃ until a sufficient amount was observed. Ultimately, 6 of the 106 nominated bacterial strains had not grown, and 100 bacterial strains which had been re-cultured were used in this study (Table 1).

Table 1. Tested Bacterial Strains

II) Instruments and software

The VITEK-MS and Microflex LT, which are MALDI-TOF MS devices available in Japan, were utilized in this study. Spectral analysis of the examined 100 sample strains was performed using the MALDI Biotyper software version 3.1 (Bruker Daltonics, Bremen, Germany). This software performs matching comparisons of spectral patterns between the sample bacteria and those from the reference databases (libraries) that each manufacturer provides and updates.

III) MALDI-TOF analysis

Aliquots of each bacteria, according to manufacturers’ instructions, were lifted from the colonies on each culture media and gently spotted on the MALDI-TOF examination plate provided by each manufacturer. Next, the cell-smear method, a procedure to disperse sample bacteria, was performed;1-μL of alpha-cyano-4-hydroxycinnamic acid matrix solution (CHCA for VITEK-MS;BioMérieux, HCCA for Microflex LT;Bruker Daltonics) was added to each sample plate. After air drying naturally, each plate was set into its respective device, and the analytical procedures were performed, yielding the mass spectra of each sample.

Table 2 characterizes the identification reliability in each MALDI-TOF MS device. Level 1 Reliability (L1R) means that the bacterial strain is identified at the species level, where the VITEK-MS shows one bacterial species with ≥ 60% reliability, or the Microflex LT shows an identification score ≥ 2.000. When the VITEK-MS shows two to four bacterial species, or the Microflex LT shows a score of 1.700 to 1.999, the tested bacterial strain is regarded as identified at Level 2 Reliability (L2R), at which species identification is unreliable but the genus is reliable enough. Therefore, even if one bacterial species is shown at L2R, the strain is regarded as identified at bacterial genus level.

When both MALDI-TOF MS devices were unable to identify a sample at L1R or L2R, a reanalysis was performed using an additional application, the on-plate formic acid extraction method7). Briefly, 1-μl of formic acid (Wako, Osaka, Japan) was used as a cell-smear solution, before adding alpha-cyano-4-hydroxycinnamic acid matrix solution.

In cases where the identification results differed between the VITEK-MS and Microflex LT, or conflicted with those of conventional protocols, the 16S ribosomal RNA (rRNA) of the sample bacteria were sequenced and each result was confirmed. Specifically, DNA was extracted from each strain, amplified by PCR using universal primers (27F and 1492R), and each product was sequenced bidirectionally to determine the nucleotide sequence. Then, the homology analysis of each 1,000 to 1,300 base pairs, including V4 to V6 regions known to be beneficial for bacterial identification8), was performed, and each species or genus was identified with 98.7% or higher homology. For analysis of the 16S rRNA sequence, the DNA Data Bank Japan (DDBJ), managed by the National Institute of Genetics (Shizuoka, Japan) was referenced.

Table 2. Identification Reliability Level of VITEK-MS and Microflex LT

IV) Calculation and statistical analysis

For the results obtained in this study, the number of sample strains identified at the species level (at L1R), identified at the genus level (at L1R and L2R), and unidentified using either MALDI-TOF MS device were calculated for the identification rates. Concordances of results between the two models were also calculated at the species and genus level, respectively. For the sample strains reanalyzed using each MALDI-TOF MS device with the formic acid extraction method, the results shown in the single application and those obtained from reanalysis were both used for analysis.

The statistical differences between the calculated results obtained from the two MALDI-TOF MS models were compared by applying McNemar’s test, including application of Yates’s correction for continuity. A p value < 0.05 was considered statistically significant.


I) Identification rates

The identification rates after each single application are shown in Table 3. Using the VITEK-MS, bacterial species was identified in 47% (47/100) at L1R, and genus in 91% (91/100) at L1R and L2R;however, 9% (9/100) remained unidentified. On the other hand, the identification rates using the Microflex LT were 71% (71/100) at L1R and 92% (92/100) at L1R and L2R, with 8% (8/100) remaining unidentified. After subsequent analysis with the formic acid extraction method, the identification rates were partly improved (Table 3). The identification rates using the VITEK-MS rose to 49% (49/100) at L1R and 96% (96/100) at L1R and L2R, with 4% (4/100) remaining unidentified. On the other hand, a total of 80% (80/100) at L1R and 95% (95/100) at L1R and L2R were identified using the Microflex LT, with 5% (5/100) remaining unidentified. Those data with the formic acid extraction method were used for the following analysis.

A comparison with conventional identification protocols follows. Of the 79 tested bacterial strains with genus-level identification through conventional protocols (47 Bacillus spp., 18 Corynebacterium spp., 8 Staphylococcus spp., 5 Streptococcus spp., and 1 Peptostreptococcus spp.), 39.2% (31/79) and 86.1% (68/79) were resolved to species-level at L1R using the VITEK-MS and the Microflex LT, respectively. Furthermore, of the 21 tested strains with neither genus nor species identification by conventional methods, 85.7% (18/21) at L1R and 95.2% (20/21) at L1R and L2R with the VITEK-MS were resolved to species or genus, respectively, while 57.1% (12/21) at L1R and 95.2% (20/21) at L1R and L2R with the Microflex LT were resolved to species or genus (Table 3).

Regarding bacterial genus, the VITEK-MS identified 4.26% (2/47) of Bacillus spp. at the species level, while the Microflex LT identified 95.7% (45/47) (p<0.001). Moreover, regarding glucose non-fermenting Gram-negative rods, the VITEK-MS identified the species in 100% (10/10), while the Microflex LT identified 40% (4/10) at the species level (p<0.005). No statistical difference was shown between the two MALDI-TOF MS devices in the identification rates at species or genus levels for bacterial strains other than the aforementioned two groups (Table 3).

Table 3. Identification Rates using MALDI-TOF MS Devices

L1R: Level 1 Reliability, meaning species-level identification

L2R: Level 2 Reliability, meaning genus-level identification

II) Identification discrepancies between the conventional protocols and MALDI-TOF MS

The respective results of the bacterial identification using the two MALDI-TOF MS devices are shown in Table 4. Of the sample bacterial strains reanalyzed using MALDI-TOF MS devices, 15 bacterial strains were identified inconsistently with the bacterial species, genus, or morphological features indicated through the conventional protocols. That is regarded as the identification discrepancy between conventional identification protocols and the MALDI-TOF MS devices (indicated by “D” on Table 4), calculated as 15% (15/100).

Among these 15 strains, 8 were identified as the same bacterial species or genus by both the VITEK-MS and Microflex LT. Three strains resolved as Corynebacterium spp. by conventional methods were reidentified as Actinomyces neuii, Arthrobacter sp., and Propionibacterium sp. Three strains thought to be Streptococcus spp. were reidentified as Actinomyces sp., Gemella morbillorum, and Weissella confusa. The other strains identified as Peptostreptococcus sp. and Glucose non-fermenting Gram-negative rod were reidentified as Staphylococcus saccharolyticus and Paenibacillus sp., respectively. Four strains were identified using one MALDI-TOF MS device, but not identified using the other (indicated by “F” on Table 4). The other 3 strains were differently identified by the two MALDI-TOF MS devices (indicated by “S” on Table 4).

Table 4. Identification Discrepancies between the Conventional Protocols and MALDI-TOF MS Devices and Concordance between VITEK-MS and Microflex LT

N: Number of samples

SubT: Subtotal

D:Discrepant identification between conventional protocols and MALDI-TOF MS

M1: Matched identification on bacterial species

M2: Consistent identification at bacterial genus level

F: Unidentified with either VITEK-MS or Microflex LT

S: Inconsistently identified by VITEK-MS and Microflex LT

III) Concordances and inconsistencies between the two MALDI-TOF MS devices

The concordance of the identification results between the two MALDI-TOF MS devices is also shown in Table 4, where the identification concordance is categorized by the bacterial strains as: M1, matched at bacterial species (at L1R);M2, consistent at bacterial genus (at L2R);F, unidentified with the VITEK-MS or unreliable identification by the Microflex LT;or S, inconsistent identification between the VITEK-MS and Microflex LT. Between the two MALDI-TOF MS devices, 26% (26/100) of M1 concordance and 88% (88/100) of M1 and M2 concordance was shown. The VITEK-MS failed to identify four bacterial strains, while the Microflex LT could not identify five strains (indicated by “F” on Table 4). Moreover, three bacterial strains were identified inconsistently using the VITEK-MS and Microflex LT (indicated by “S” on Table 4), as mentioned above.

IV) Confirmation of the discrepant identification results by the 16S rRNA sequencing

Of the nine strains that were reidentified using one MALDI-TOF MS device but not identified using the other, the results of four strains were inconsistent with those by conventional protocols, as mentioned above. Therefore, 16S rRNA sequencing was performed additionally to confirm these discrepancies, and it was verified that three of those four identification results using the MALDI-TOF MS devices were correct (Table 5). The one other strain, which came out as Corynebacterium sp. through conventional protocols but was reidentified as Paenibacillus durus using the VITEK-MS at L1R, was still not resolved by 16S rRNA analysis because the nucleotide sequence of this strain could not be identified by sequencing. Samples for 16S rRNA sequencing were taken from frozen stocks;therefore, bacterial contamination or failure to preserve strains were considered as possible cause of this failure.

On the other hand, the identification results of three strains were different between the VITEK-MS and Microflex LT, and either identification result was inconsistent with that of conventional identification protocols. One of those, thought in advance to be a glucose non-fermenting Gram-negative rod, was reidentified as Mycobacterium kansasii at L1R using the VITEK-MS, versus Herbaspirillum huttiense at L2R using the Microflex LT. This strain was confirmed as Herbaspirillum sp. by 16S rRNA sequencing. The other strain, a Gram-positive rod according to conventional identification protocols, was resolved as Paenibacillus durus at L1R with the VITEK-MS, versus Mycobacterium peregrinum at L2R using the Microflex LT. This strain was confirmed as Mycolicibacterium sp., which had been categorized among Mycobacterium spp. before 2018, by 16S rRNA sequencing. Another strain, thought to be an anaerobic Gram-negative rod by conventional methods, was identified as Streptococcus pseudopneumoniae with the VITEK-MS, versus Bilophila sp. using the Microflex LT. This strain was confirmed as Bilophila wadsworthia by 16S rRNA sequencing (Table 5).

Table 5. Confirmation by 16S rRNA Sequencing for Discrepantly Identified Bacterial Strains


Blood culture is considered as the gold standard for diagnosis of BSIs. Immediate administration of effective antibiotics is essential for the appropriate treatment of BSIs caused by bacteria;therefore, the importance of blood culture testing is supported by the rapid and accurate identification of the pathogenic bacteria2). Clinical superiority of MALDI-TOF MS devices compared to conventional morphological and biochemical identification procedures had been shown in previous studies, and also shown in the present study3-5). Both MALDI-TOF MS models, validated in our study and available in Japan, identified 95-96% of the tested bacterial strains at L1R and L2R, and resolved 49-80% of the strains at L1R that were not identified through conventional protocols. The mechanism of MALDI-TOF MS is to ionize sample microbes and to deduce their material composition based on flight time. The data are collected worldwide and compiled into spectral databases, with which homological analysis is performed to identify the microbes. This procedure is fundamentally different from conventional protocols to identify pathogenic bacteria, in that minute differences in the composition of each microbe can be recognized using MALDI-TOF MS, which operates independently of human senses. That is considered as the reason why MALDI-TOF MS devices achieve more accurate bacterial identification than conventional morphological and biochemical procedures.

As for the sample preparation for the MALDI-TOF MS, some of the bacterial samples were not well analyzed using the simple cell-smear method, a standard preparation for MALDI-TOF MS devices, whereas the additional formic acid extraction method enabled efficient analysis. This is independent of the models and attributed to poor ionization of the tested bacteria because of their cell wall structures and/or capsular formations. Formic acid breaks the bacterial cell wall, allowing proteins to be extracted, which has been reported to improve ionization in MALDI-TOF MS technology and increase its reliability, as compared to the single cell-smear procedure 9). If prior Gram-stain and/or colony morphology predicts that the targeted bacterial strain has robust cell walls, such as yeast-like fungus or Corynebacterium spp., the direct induction of the formic acid extraction method may improve laboratory throughput.

A previous study reported that genetically very related bacterial species, showing high homology by 16S rRNA sequencing, were difficult to identify using MALDI-TOF MS devices10). In the present study, the concordance rates between the VITEK-MS and Microflex LT reached more than 90%, although each model yielded slightly different identification results. As an example, the candidate bacterial species that the VITEK-MS showed for Bacillus spp. contained two or three genetically similar species, while the Microflex LT often showed just one species. According to a recent report, the VITEK-MS exhibited poor identification properties for differentiating among Bacillus cereus, Bacillus thuringiensis, and Bacillus mycoides, as well as between Bacillus subtilis and Bacillus amyloliquefaciens11). This may be due to the low concordance rates at L1R with the VITEK-MS;this is supported by the current study. On the other hand, the identification rate at L1R for glucose non-fermenting Gram-negative rods resulted in a low profile using the Microflex LT, while that using the VITEK-MS resulted in a high profile. The Microflex LT analyzes raw spectral data of bacterial strains for identification, while the VITEK-MS utilizes data correction for bacterial identification. Because the principle of MALDI-TOF MS to determine the sample composition from the flight time of ions is the same for both models, this difference in analytical approaches between the two MALDI-TOF MS devices may affect the differing identification rates at L1R12,13).

The concordance rate between the two MALDI-TOF MS models was 88% (88/100) for the bacterial strains identified at L1R and L2R, though nine slow-growing bacterial strains remained unidentified using either model. Those bacterial strains were cultivated until visible colonies appeared, to be examined as target materials. Therefore, MALDI-TOF MS technology may not be as good at identification of non-fresh strains and/or those of insufficient quantity. In addition, inadequate preservation of some bacterial strains may have contributed to their non-identification in the present study. A recent report discussed unique approaches in which microorganisms contained in positive blood culture bottles were directly identified by means of MALDI-TOF MS devices14). If we had been able to test fresh bacterial strains cultivated directly from the positive blood cultures, we may have been able to identify those bacterial strains.

Of all the strains examined in the current study, three strains were differently identified using the VITEK-MS and Microflex LT, which were confirmed by 16S rRNA sequencing (Table 4, 5). The three strains were verified to be Bilophila wadsworthia, Herbaspirillum sp., and Mycolicibacterium sp., respectively, consistent with the results from the Microflex LT in all three cases. It has been reported that Mycobacterium spp. requires specialized treatment before applying identification procedures15), however, the Microflex LT worked without such specialized pre-test treatments. On the other hand, another strain was confirmed to be Bilophila wadsworthia, one of the anaerobic Gram-negative rods. As discussed in previous reports16,17), the data of anaerobic bacteria provided for MALDI-TOF MS analysis is still incomplete and variable, which may have some effect on the identification of anaerobic bacteria. This outcome may be a coincidence, however, since a technical difference between the two MALDI TOF-MS devices, namely, whether to use raw or processed data sets, is probably one of the factors, as mentioned above. Furthermore, since those bacterial strains are rarely detected through blood culture alone, further enhancement of the reference database for each MALDI-TOF MS device may enable more correct bacterial identification.

With regard to comparing results from conventional methods with those using MALDI-TOF MS devices, the discrepancy rate was 15% (15/100) as shown in Table 4. Of those 15 strains, 53.3% (8/15) were identified as the same bacterial species or genus using both of the MALDI-TOF MS devices, which suggests that MALDI-TOF MS techniques could provide more beneficial information in the clinical settings than conventional identification protocols. Of the two tested bacterial strains that had been identified as Bacillus spp. through conventional protocols, one was reidentified as Lysinibacillus fusiformis using the VITEK-MS at L1R, and the other as Paenibacillus chitinolyticus using the Microflex LT at L2R, both of which were verified by 16S rRNA sequencing (Table 5). These bacterial species, Lysinibacillus sp. and Paenibacillus sp., were classified as Bacillus spp. in the past, and were known as difficult species to identify through conventional protocols18,19). The bacterial strains identified as Corynebacterium spp. through conventional protocols were reidentified as Actinomyces neuii, Arthrobacter cumminsii, or Propionibacterium acnes by both MALDI-TOF MS devices. These bacterial species show positive Gram-stainability and are classified as Coryneform bacteria or Coryneform group generally. It had been known that the only way to identify these strains was by Gram-stain, suggesting that there might be a limitation for the identification of these strains by conventional methods. The bacterial strains identified as Streptococcus spp. by conventional methods were reidentified as Actinomyces europaeus, Gemella morbillorum, or Weissella confusa by MALDI-TOF MS. Gemella morbillorum is classified as a Gram-positive cocci, and its morphological features are very similar to those of Streptococcus spp. Gemella morbillorum was previously classified under the genus Streptococcus20), and was also difficult to identify using conventional protocols. Weissella confusa and Actinomyces europaeus are Gram-positive coccobacilli or coccoid rods21,22), and are also difficult to identify through conventional protocols because they form very small colonies. Peptostreptococcus spp., identified using the RapID ANA II System with limited reliability, was reidentified as Staphylococcus saccharolyticus by both MALDI-TOF MS devices. Of the two bacterial strains identified as glucose non-fermenting Gram-negative rods by conventional methods, one was reidentified as Paenibacillus sp. by both MALDI-TOF MS devices, and the other was reidentified as Mycobacterium kansasii using the VITEK-MS, versus Herbaspirillum huttiense using the Microflex LT. This strain was subsequently identified as Herbaspirillum sp. by 16S rRNA sequencing (Table 5). Paenibacillus spp. and Mycobacterium spp. can be either Gram negative or Gram positive, and often tend to be identified as glucose non-fermenting Gram-negative rods using a MicroScan WalkAway System, though such results exhibit poor reliability. The strains reported as an anaerobic Gram-negative rod through conventional protocols resulted in identification discrepancies between the two MALDI-TOF MS devices, where the VITEK-MS identified it as Streptococcus pseudopneumoniae at L1R, versus the Microflex LT as Bilophila sp. This strain was also verified to be Bilophila wadsworthia by 16S rRNA sequencing. The previous studies had reported that the VITEK-MS was not good at identification of anaerobic bacteria23), which is consistent with this study.

In previous studies comparing MALDI-TOF MS devices, there were not such large differences between the bacterial identification rates of the VITEK-MS (92.3-99.8%) and Microflex LT (95.5-97.8%) at bacterial genus and species level23-25). Although we used the bacterial strains whose species were unidentified by conventional methods, almost the same results were obtained in this study, which suggests the convenience of MALDI-TOF MS devices for bacterial identification. Considering the results according to each bacterial species, as mentioned in previous studies, the VITEK-MS tended not to be good at identifying Bacillus spp., while the Microflex LT tended to be weak at identifying glucose non-fermenting Gram-negative rods24-26). Moreover, both MALDI-TOF MS devices had been reported to have difficulty identifying anaerobic bacteria23).

The MALDI-TOF MS technology for the identification of bacterial species was validated and demonstrated to be easy to use and time-saving in this study. These features contribute to the management of BSIs and other serious infectious diseases, since earlier identification of pathogenic bacteria facilitates effective treatment. As mentioned above, each MALDI-TOF MS model has its weaknesses identifying particular bacteria. In a situation where both the VITEK-MS and Microflex LT are available, the proper use of these models is considered: the VITEK-MS for glucose non-fermenting bacteria, and the Microflex LT for Gram-positive rods. However, few institutions or laboratories are actually equipped with both devices. Therefore, it seems important to utilize the MALDI-TOF MS devices based on the characteristics of each model. As a way to complement each weakness, it is recommended to perform other identification methods such as 16S rRNA analysis in parallel. It is also necessary to strive to enhance the laboratory knowledge and skills related to general microbiological investigations. Further accumulation of the data for MALDI-TOF MS analysis will help with more accurate identification of pathogenic bacteria.

© 2020 The Fukushima Society of Medical Science

This article is licensed under a Creative Commons [Attribution-NonCommercial-ShareAlike 4.0 International] license.