Unsupervised spectral feature selection algorithms for high dimensional data

It is a significant and challenging task to detect the informative features to carry out explainable analysis and build an interpretable AI system for high dimensional data, especially for those with very small number of samples while without any label information. Unsupervised feature selection algorithms are the right way to deal with this challenge and realize the task, especially in big data era. However, the available unsupervised feature selection approaches usually cannot precisely identify the most discriminative features from high dimensional data with small number of samples.
To address the aforementioned challenges, a research team led by Juanying XIE published their new research on 15 Oct 2023 in Frontiers of Computer Science co-published by Higher Education Press and Springer Nature.
The team proposes two novel unsupervised spectral feature selection algorithms, which group features into clusters using advanced Self-Tuning spectral clustering algorithm based on local standard deviation, guaranteeing the global optimal feature clusters could be detected as far as possible. The entropy-based and cosine-similarity-based feature ranking techniques are, respectively, proposed, so that the representative feature from each cluster could be detected out to comprise the feature subset on which an explainable classification system will be built. This guarantees that the detected features are representative and independence each other as far as possible. The extensive experiments and rigorous statistical tests demonstrate that these unsupervised spectral feature selection algorithms are superior to the peer ones in comparison. They detected features having strong discriminative capabilities in downstream classifiers for omics data, such that the AI system built on them would be reliable and explainable, making it possible to build a transparent and trustworthy medical diagnostic system from an interpretable AI perspective.
For the future direction, one is to study the general way for finding an appropriate parameter of the advanced Self-Tuning spectral clustering based on local standard deviation. Another is to reduce the computing cost when detecting the optimal feature subset of very very high dimensionality data, such as SNP data.