A Study on Regression and Loss Functions for Multiscale k-Nearest Neighbour

Abstract

k-nearest neighbour (k-NN) takes label average over a query ball, whose radius r_k increases with larger k, and the non-zero radius results in a bias of the k-NN estimator. To reduce the bias, multiscale k-NN (MS-k-NN) first solves ordinary least squares (OLS) to predict the k-NN estimator at some points k=k₁, k₂, ..., k_V from even-degree polynomials of the radius r_k, and extrapolates the estimator to r=0. However, there remain two practical problems: (i) The polynomial used for extrapolation is derived from asymptotic theory; in finite-sample situations, the MS-k-NN estimator with even-degree polynomials is not necessarily restricted to a proper range [0,1]. (ii) OLS implicitly assumes the independence of the k-NN estimators at k=k₁, k₂, ..., k_V, whereas the estimators utilizing some same labels are dependent. To solve these problems, we propose employing sigmoid-based functions and generalized least squares. We also propose local radial logistic regression (LRLR), which is inspired by MS-k-NN.