The robust controller is often required for suppressing modeling errors and disturbances. In order to design the robust controller analytically, the H∞ control problem and the µ-synthesis are proposed as the model-based controller design techniques. These two methods need the nominal plant model and the uncertainty. However, it is difficult to obtain the accurate plant model with limited complexity. The inappropriate plant model leads to large uncertainty. As a result, the designed controller is conservative. This paper proposes designing the robust controller with only frequency responses. In addition, the proposed method designs the controller automatically by the numerical optimization. In the method, the robust performance condition is addressed as the design problem. The robust performance condition is formulated as a iterative convex optimization using the linear approximation by the Taylor expansion. However, since the Taylor expansion is used for the formulation, the initial controller is needed and the designed controller depends on the initial controller. To solve the problem, the particle swarm optimization (PSO) is introduced to obtain a reasonable initial controller. The PSO reduces dependence on initial controllers because many initial controllers are given randomly as candidates of the solution. Simulation and experimental results are shown to confirm the effectiveness of the proposed method.