The development of open-domain conversational systems is difficult since user utterances are too flexible for such systems to respond properly. To address this flexibility, previous research on conversational systems has selected system utterances from web articles based on word-level similarity with user utterances; however, the generated utterances, which originally appeared in different contexts from the conversation, are likely to contain irrelevant information with respect to the input user utterance. To leverage the variety of web corpus in order to respond to the flexibility and suppress the irrelevant information simultaneously, we propose an approach that generates system utterances with two strongly related phrase pairs: one that composes the user utterance and another that has a dependency relation to the former. By retrieving the latter one from the web, our approach can generate system utterances that are related to the topics of user utterances. We examined the effectiveness of our approach with following two experiments. The first experiment, which examined the appropriateness of response utterances, showed that our proposed approach significantly outperformed other retrieval and rule-based approaches. The second one was a chat experiment with people, which showed that our approach demonstrated almost equal performance to a rule-based approach and outperformed other retrieval-based approaches.