Recent developments in non-invasive neuroimaging technologies have allowed for research on a brain-computer interface (BCI). A BCI is a system that operates a machine using only brain activity. Thus, a BCI is expected to help people with disabilities, assisting them in communicating or making decisions. In this study, we focused on assisting with spelling. Therefore, our purpose in conducting this study was to develop a BCI system for assisting with spelling. To develop our system, we used near-infrared spectroscopy (NIRS) to measure brain activity in the prefrontal cortex region. Mental arithmetic was the method of stimulating the activity of the brain. As a result of the experiment, the individual variation was observed in each participant, in the channel and timing of activating the brain. Therefore, the method by which the system automatically selects learning and classification data was introduced. In addition, we proposed the stimulus presentation method of “bit form” to replace the “matrix form” used commonly by previous researches. Then the classification accuracy was verified offline. As a result of this analysis, classification accuracy enhancement to 78.47% has been shown by support vector machine (SVM). Though additional examination is needed, the future potential of this system using NIRS was demonstrated.