This paper aimed at quantifying the actual number of vacant houses and their location characteristics in mountain villages. A survey was conducted in a community in Nagawa Town, Nagano Prefecture, where the population has been declining. There is common assumption that the rate of vacant houses, i.e., whether or not to leave a community, is influenced by the socioeconomic location of the community, such as employment opportunities and convenience of living. This paper hypothesized that whether people continue coming and going after leaving a community may be influenced by factors separate from convenience, such as attachment to the community. The result showed that convenience had a significant impact on the percentage of vacant houses, but the inconvenient location was not a major disincentive to “continuing to be involved” in the community.