A number of studies have focused largely on whether learners with and without experience of English language instruction in elementary school or somewhere in early childhood are different in terms of a single factor such as English language proficiency and attitude/motivation. However, little empirical research has been conducted to clarify how multiple learner features are correlated with English proficiency and affective factors. This study investigates effects of multiple learner features, some of which are especially relevant to early English language learning, on English language proficiency and affective factors at junior high school level. The results show that English proficiency has a relatively strong relationship with the length of learning, attitude toward learning English, and motivation toward learning English for academic purposes, whereas the major affective factor, i.e., attitude toward learning English, shows a strong relationship with English proficiency, the age of onset, and English learning experience at cram school. Attitude toward learning Japanese, however, does not have any strong relationship with other target variables. These findings suggest that it should be important to take multiple relevant factors into consideration when implementing English language teaching at earlier ages.