A smart electric wheelchair used as a personal mobility tool (PMT) has been proposed. To reduce an operational load, the PMT is integrated with autonomous mobile robot navigation and human operation input. It is able to reach a destination autonomously based on a directional input from a user. A destination of the PMT is selected from the user's directional input. Therefore, accuracy of destination selection depends on the user's input. To realize stress-free and efficient movement of the PMT, it is required to select a suitable destination based on inaccurate human input. In this paper, we propose a selection method for a suitable destination using an environment map, which describes not only static obstacles such as wall and furniture but also areas for human activity. To realize this, this paper also proposes an environmental map including areas for human activity. An experiment is conducted to evaluate the proposed method of destination selection based on performance of the PMT operation and mental workload of subjects.