This paper addresses a framework in dialog systems that can understand users' speech depending on location during outdoor activities. Depending on the outdoor context, systems should be able to correctly interpret spoken sentences in changing locations. A promising approach is to combine the knowledge of users' goal and plan with information of their position since their speech is normally dependent on their goal and plan, which can be related with their location in an outdoor activity. To test this approach, a user support system called G-Assist has been developed for mobile PCs. G-Assist provides pieces of advice to beginner golf players in response to their questions via voice on the spot. By defining the associations of users' goal and plans and spaces in the knowledge base, the system dynamically activates the corresponding part of a goal and plan with respect to the users' position or a location provided by the GPS during the dialog. To interpret a spoken sentence, the system integrates bottom-up information based on analysis of the sentence with top-down information derived from the users' goal and plan definitions. The framework is useful in analyzing incomplete sentences in which information dependent on the location is omitted. Furthermore, the top-down information is utilized to revise the result of voice recognition so that a recognized word can be replaced with another word of similar sound highly related to activating goal and plans.