It is desirable for a container crane to operate smoothly and quickly. For this purpose, the control system of a container crane should be capable of anti-sway control for suppressing vibrations. A vision sensor system is often used to detect the sway angle. However, since a control system with a vision sensor has a delay time when determining the angle, it sometimes leads to the deterioration of the control performance owing to the delay time. In order to overcome this problem, this paper proposes a new anti-sway crane control system based on a dual-state observer with sensor-delay correction. However, because of nonlinear friction in the crane, the estimation accuracy achieved by using the observer is poor. To overcome this problem, this paper proposes a disturbance observer considering friction disturbance. The control performance and effectiveness of the proposed robust control system based on the estimated information are confirmed to be satisfactory from experimental results.