It is essential for robot to correctly monitor and track human workers' hands or fingers in hybrid cell where human workers and robots work together to accomplish an assembly task. In this paper, a sensor system setup for hybrid cell is introduced where a kinect on top and a 3-D acceleration and gyro sensor on human workers' hands are adopted. An effective algorithm to track the human workers' hands and fingers trajectory in 3-D space is proposed. The experimental results verify the effectiveness of this sensor system.