Abstract
Let Let $\mathcal{O}$ be a nilpotent orbit in $\mathfrak{so}(p,q)$ under the adjoint action of the full orthogonal group $\mathrm{O}(p,q)$. Then the closure of $\mathcal{O}$ (with respect to the Euclidean topology) is a union of $\mathcal{O}$ and some nilpotent $\mathrm{O}(p,q)$-orbits of smaller dimensions. In an earlier work, the first author has determined which nilpotent $\mathrm{O}(p,q)$-orbits belong to this closure. The same problem for the action of the identity component $\mathrm{SO}(p,q)^0$ of $\mathrm{O}(p,q)$ on $\mathfrak{so}(p,q)$ is much harder and we propose a conjecture describing the closures of the nilpotent $\mathrm{SO}(p,q)^0$-orbits. The conjecture is proved when $\min(p,q) \le 7$.
Our method is indirect because we use the Kostant-Sekiguchi correspondence to translate the problem to that of describing the closures of the unstable orbits for the action of the complex group $\mathrm{SO}_p(\boldsymbol{C})\times\mathrm{SO}_q(\boldsymbol{C})$ on the space $M_{p,q}$ of complex $p\times q$ matrices with the action given by $(a,b)\cdot x=axb^{-1}$. The fact that the Kostant-Sekiguchi correspondence preserves the closure relation has been proved recently by Barbasch and Sepanski.
