Abstract
Phase transition of frustrated Heisenberg antiferromagnet on the three-dimensional layered-triangular lattice is studied via a symmetry argument and Monte Carlo simulations. A symmetry argument shows that the order-parameter space of this model is isomorphic to the three-dimensional rotation group SO(3) or the projective space P3, different from those of the corresponding nonfrustrated systems. The possibility is suggested that the system exhibits a novel type of phase transition characterized by its SO(3) symmetry. Extensive Monte Carlo simulations indicate the occurrence of a single continuous transition. By applying the finite-size scaling, respective exponents are estimated as α\simeq0.4, β\simeq0.25, γ\simeq1.1 and ν\simeq0.53. These values differ considerably from those for the standard nonfrustrated systems strongly suggesting a phase transition belonging to a new universality class. Several systems, including the superfluid A phase of helium 3, certain types of helical or canted magnets, are predicted to belong to the same universality class. Possible consequences for experiments are discussed.
