Dissertation/ Thesis
Probabilistic Approaches for Deep Learning: Representation Learning and Uncertainty Estimation
| Title: | Probabilistic Approaches for Deep Learning: Representation Learning and Uncertainty Estimation |
|---|---|
| Authors: | Park, Yookoon |
| Publication Year: | 2024 |
| Collection: | Columbia University: Academic Commons |
| Subject Terms: | Computer science, Deep learning (Machine learning), Neural networks (Computer science), Bayesian statistical decision theory, Probabilities--Computer programs |
| Description: | In this thesis, we present probabilistic approaches for two critical aspects of deep learning: unsupervised representation learning and uncertainty estimation. The first part of the thesis focuses on developing a probabilistic method for deep representation learning and an application of representation learning on multimodal text-video data. Unsupervised representation learning has been proven effective for learning useful representations of data using deep learning and enhancing the performance on downstream applications. However, current methods for representation learning lack a solid theoretical foundation despite their empirical success. To bridge this gap, we present a novel perspective for unsupervised representation learning: we argue that representation learning should maximize the effective nonlinear expressivity of a deep neural network on the data so that the downstream predictors can take full advantage of its nonlinear representation power. To this end, we propose our method of neural activation coding (NAC) that maximizes the mutual information between activation patterns of the encoder and the data over a noisy communication channel. We show that learning for a noise-robust activation code maximizes the number of distinct linear regions of ReLU encoders, hence maximizing its nonlinear expressivity. Experiment results demonstrate that NAC enhances downstream performance on linear classification and nearest neighbor retrieval on natural image datasets, and furthermore significantly improve the training of deep generative models. Next, we study an application of representation learning for multimodal text-video retrieval. We reveal that when using a pretrained representation model, many test instances are either over- or under-represented during text-video retrieval, hurting the retrieval performance. To address the problem, we propose normalized contrastive learning (NCL) that utilizes the Sinkhorn-Knopp algorithm to normalize the retrieval probabilities of text and video instances, thereby ... |
| Document Type: | thesis |
| Language: | English |
| DOI: | 10.7916/p28h-6109 |
| Availability: | https://doi.org/10.7916/p28h-6109 |
| Accession Number: | edsbas.617B9FB7 |
| Database: | BASE |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://doi.org/10.7916/p28h-6109# Name: EDS - BASE (ns324271) Category: fullText Text: View record from BASE |
|---|---|
| Header | DbId: edsbas DbLabel: BASE An: edsbas.617B9FB7 RelevancyScore: 867 AccessLevel: 3 PubType: Dissertation/ Thesis PubTypeId: dissertation PreciseRelevancyScore: 866.983581542969 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Probabilistic Approaches for Deep Learning: Representation Learning and Uncertainty Estimation – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Park%2C+Yookoon%22">Park, Yookoon</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2024 – Name: Subset Label: Collection Group: HoldingsInfo Data: Columbia University: Academic Commons – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+science%22">Computer science</searchLink><br /><searchLink fieldCode="DE" term="%22Deep+learning+%28Machine+learning%29%22">Deep learning (Machine learning)</searchLink><br /><searchLink fieldCode="DE" term="%22Neural+networks+%28Computer+science%29%22">Neural networks (Computer science)</searchLink><br /><searchLink fieldCode="DE" term="%22Bayesian+statistical+decision+theory%22">Bayesian statistical decision theory</searchLink><br /><searchLink fieldCode="DE" term="%22Probabilities--Computer+programs%22">Probabilities--Computer programs</searchLink> – Name: Abstract Label: Description Group: Ab Data: In this thesis, we present probabilistic approaches for two critical aspects of deep learning: unsupervised representation learning and uncertainty estimation. The first part of the thesis focuses on developing a probabilistic method for deep representation learning and an application of representation learning on multimodal text-video data. Unsupervised representation learning has been proven effective for learning useful representations of data using deep learning and enhancing the performance on downstream applications. However, current methods for representation learning lack a solid theoretical foundation despite their empirical success. To bridge this gap, we present a novel perspective for unsupervised representation learning: we argue that representation learning should maximize the effective nonlinear expressivity of a deep neural network on the data so that the downstream predictors can take full advantage of its nonlinear representation power. To this end, we propose our method of neural activation coding (NAC) that maximizes the mutual information between activation patterns of the encoder and the data over a noisy communication channel. We show that learning for a noise-robust activation code maximizes the number of distinct linear regions of ReLU encoders, hence maximizing its nonlinear expressivity. Experiment results demonstrate that NAC enhances downstream performance on linear classification and nearest neighbor retrieval on natural image datasets, and furthermore significantly improve the training of deep generative models. Next, we study an application of representation learning for multimodal text-video retrieval. We reveal that when using a pretrained representation model, many test instances are either over- or under-represented during text-video retrieval, hurting the retrieval performance. To address the problem, we propose normalized contrastive learning (NCL) that utilizes the Sinkhorn-Knopp algorithm to normalize the retrieval probabilities of text and video instances, thereby ... – Name: TypeDocument Label: Document Type Group: TypDoc Data: thesis – Name: Language Label: Language Group: Lang Data: English – Name: DOI Label: DOI Group: ID Data: 10.7916/p28h-6109 – Name: URL Label: Availability Group: URL Data: https://doi.org/10.7916/p28h-6109 – Name: AN Label: Accession Number Group: ID Data: edsbas.617B9FB7 |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsbas&AN=edsbas.617B9FB7 |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.7916/p28h-6109 Languages: – Text: English Subjects: – SubjectFull: Computer science Type: general – SubjectFull: Deep learning (Machine learning) Type: general – SubjectFull: Neural networks (Computer science) Type: general – SubjectFull: Bayesian statistical decision theory Type: general – SubjectFull: Probabilities--Computer programs Type: general Titles: – TitleFull: Probabilistic Approaches for Deep Learning: Representation Learning and Uncertainty Estimation Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Park, Yookoon IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2024 Identifiers: – Type: issn-locals Value: edsbas – Type: issn-locals Value: edsbas.oa |
| ResultId | 1 |