In order to clarify the effect of dislocations that are statistically stored dislocations (SSDs) and geometrically necessary dislocations (GNDs) on the characteristic length dependent micro-to macroscopic deformation behavior of polycrystalline metals, finite element simulations based on a crystal plasticity theory accounting for SSDs and GNDs have been performed for plane stress polycrystalline metals with different grain sizes. The dislocation lines constructed based on the edge and screw components of GNDs on the slip planes clarify that the grain boundary as well as sub-grain boundary act as the obstacles to dislocation motion and impede easy motion of dislocations. The density of SSDs and GNDs increases as the gain size decreases that yields the increase in the deformation resistance in macroscopic scale. The mean glide path distance and annihilation distance of dislocations strongly affect the micro-to macroscopic deformation behavior of polycrystalline metals. Hall-Petch-like relations between the macroscopic resistance of deformation and grain size are observed.