Evolutionary Representation Learning of Structured Multi-label Data

Multi-label representation learning is a machine-learning paradigm that concerns the active presence of multiple class labels for an instance of data. Multi-label research has broadly focused on two aspects of representation learning: the first aims to improve the accuracy of a set of class labels predicted by a multi-label classifier by removing irrelevant or redundant features, known as feature selection, and the second by improving the optimisation behaviour, and the learning objective, of the multi-label model. Multi-label feature selection has naturally tended toward efficient and effective methods such as sparsity-based feature selection, which is a class of gradient-based matrix optimisation techniques. However, the gradient-based optimisation process usually introduces critical convergence and consistency issues, especially due to the large search space of features selection, and when multi-label classification performance metrics are non-convex, non-differentiable, or in conflict. Moreover, despite the state-of-the-art performance of deep learning for multi-label classification, gradient-based learning introduces consistency issues in goal alignment between surrogate objective functions and the desired multi-label performance metric. Lastly, the prevalence of deep learning for tabulated multi-label models also introduces security and safety risks from malicious adversarial attacks.

This thesis begins by presenting a pioneering work on sparsity-based multi-label feature selection using evolutionary methods for differentiation-free optimisation. The proposed methods are designed to detect various critical feature-to-feature interactions that can improve classification performance. This thesis finds that the proposed approach can consistently achieve better classification performance than several widely used traditional gradient-based methods on structured (tabulated) multi-label data.

This thesis continues by expanding on the pioneering work in enabling differentiation-free optimisation by introducing a highly novel and comprehensive fitness function to detect feature-to-feature, feature-to-label, and label-to-label interactions simultaneously. A new initialisation method is proposed to complement the aforementioned fitness function. Empirical results on several benchmark tabulated multi-label datasets place the proposed method as state-of-the-art in comparison to contemporary multi-label feature selection methods.

This thesis also presents a new embedded multi-label feature selection method to optimise multiple conflicting multi-label performance metrics and the number of selected features simultaneously, which forms a many-objective optimisation problem. Decomposition techniques are developed to reduce search and objective dimensionality and a novel differentiation-free local search framework is proposed. The results indicate that the proposed method can find diverse fronts also containing high-quality solutions in comparison to benchmark many-objective and state-of-the-art traditional gradient-based methods on multi-label tabulated data.

This thesis presents a groundbreaking learning objective to optimise the parameters of a neural network by directly using several non-convex or discontinuous multi-label performance metrics. The efficacy of the proposed method is first validated by our theoretical proof which underlines the consistency of the method. Empirical evidence supports our theory by demonstrating the state-of-the-art classification performance on tabulated multi-label data of the proposed method in comparison to well-known deep learning-based models.

Finally, this thesis investigates the crucial issues of adversarial robustness of deep learning-based methods in tabulated multi-label learning. The proposed method aims to balance attack success, surrogate model robustness, and attack concealability in a many-objective black-box attack scenario. The unequivocal attack success and conventional concealability of the attack shed light on the vulnerability of existing research.