We show that parabolic Kazhdan-Lusztig polynomials of type $A$ compute the decomposition numbers in certain Harish-Chandra series of unipotent characters of finite groups of Lie types $B$, $C$ and $D$ over a field of non-defining characteristic $\ell$. Here, $\ell$ is a ``unitary prime" -- the case that remains open in general. The bipartitions labeling the characters in these series are small with respect to $d$, the order of $q$ mod $\ell$, although they occur in blocks of arbitrarily high defect. Our main technical tool is the categorical action of an affine Lie algebra on the category of unipotent representations, which identifies the branching graph for Harish-Chandra induction with the $\widehat{\mathfrak{sl}}_d$-crystal on a sum of level $2$ Fock spaces. Further key combinatorics has been adapted from Brundan and Stroppel's work on Khovanov arc algebras to obtain the closed formula for the decomposition numbers in a $d$-small Harish-Chandra series.