This paper describes a method to track moving objects with multiple cooperative visual attention windows. The method is based on an implementation of transputer-based multi-processor vision system with a correlation processor. The vision system provides a large number of attention windows, each of which can be tracked using fast correlation operation performed in a special LSI MEP. On the top of the system, we built a software manager which enables to program image processing, motion control of a window, and cooperative control of windows. The cooperation is achieved using constraints, which have tree-structure, between windows. Cooperation of multiple attentions enables to track structured objects in natural scene. As examples, human arm tracking and passing car tracking are shown.