Accurately calculating the transmission-line constants of underground cables is important for various types of power system simulations such as electromagnetic transient, power flow, and transient stability simulations. The transmission-line constants of cables installed in a tunnel must be calculated taking the skin and proximity effects into account, since the cables are spaced closely to each other and also to a wall face of the tunnel. However, the conventional method, which is capable of considering the skin and proximity effects, has been restricted to circular cross-section conductors such as cables and a circular tunnel. Therefore, the method cannot handle a tunnel with a rectangular cross section. In this paper, a method for calculating the transmission-line constants of a conductor system in which both circular and rectangular cross-section conductors present in the same space is proposed. Then, the walls of a rectangular tunnel are represented by eight rectangular conductors, and cables represented by circular conductors are placed inside the eight rectangular conductors. In this paper, the proposed method is verified by experiments using a conductor system in which three circular conductors are arranged in a rectangular conductor. Calculated results obtained by the proposed method agree well with the measured results.