Bibliographic Details
| Title: |
Hierarchical text classification with transformer-based language models |
| Authors: |
Du Toit, Jaco |
| Contributors: |
Dunaiski, Marcel, Stellenbosch University. Faculty of Science. Dept. of Computer Science. |
| Publisher Information: |
Stellenbosch University |
| Publication Year: |
2024 |
| Collection: |
Stellenbosch University: SUNScholar Research Repository |
| Subject Terms: |
Natural language processing (Computer science) -- Data processing, Hierarchical text classification, Computational linguistics, UCTD |
| Description: |
Thesis (MSc)--Stellenbosch University, 2024. ; ENGLISH ABSTRACT: Hierarchical text classification (HTC) is a natural language processing (NLP) task which has the objective of classifying text documents into a set of classes from a structured class hierarchy. For example, news articles can be classified into a hierarchical class set which comprises broad categories such as “Politics” and “Sport” in higher-levels with associated finer-grained categories such as “Europe” and “Cycling” in lower-levels. In recent years many different NLP approaches have been significantly improved through the use of transformer-based pre-trained language mod- els (PLMs). PLMs are typically trained on large amounts of textual data through self-supervised tasks such that they acquire language understanding capabilities which can be used to solve various NLP tasks, including HTC. In this thesis, we propose three new approaches for leveraging transformer-based PLMs to improve classification performance on HTC tasks. Our first approach formulates how hierarchy-aware prompts can be applied to discriminative language models such that it allows HTC tasks to scale to problems with very large hierarchical class structures. Our second approach uses label-wise attention mechanisms to obtain label-specific document repre- sentations which are used to fine-tune PLMs for HTC tasks. Furthermore, we propose a label-wise attention mechanism which splits the attention mecha- nisms into the different levels of the class hierarchy and leverages the predic- tions of all ancestor levels during the prediction of classes at a particular level. The third approach combines features extracted from a PLM and a topic model to train a classifier which comprises convolutional layers followed by a label- wise attention mechanism. We evaluate all three approaches comprehensively and show that our first two proposed approaches obtain state-of-the-art per- formances on three HTC benchmark datasets. Our results show that the use of prompts and label-wise attention ... |
| Document Type: |
thesis |
| File Description: |
xvi, 128 pages : illustrations (some color); application/pdf |
| Language: |
English |
| Relation: |
https://scholar.sun.ac.za/handle/10019.1/130642 |
| Availability: |
https://scholar.sun.ac.za/handle/10019.1/130642 |
| Rights: |
Stellenbosch University |
| Accession Number: |
edsbas.E2179A37 |
| Database: |
BASE |