The primary cancer sites for the brain metastasis site (BM) should be identified for selection of optimal treatment approaches. However, it would be difficult to identify the primary cancer for some patients even using biopsy. Therefore, more reliable identification approaches of the primary cancer sites are indispensable for decision making of appropriate treatments. This study attempts to develop an identification approach of radiomic features based on the Hessian index of differential topology. 309 patients (610 T1-weighted contrast-enhanced magnetic resonance images) who have BM were chosen for calculating image features and constructing a light gradient boosting machine model for identification of primary cancer sites. The proposed model achieved higher AUCs of 0.77 and 0.66 in a training and test, respectively, with the highest robustness index of 5.14 than other conventional models. The proposed approach could have a potential for identifying primary cancer sites, but it should be improved.