Dissertation/ Thesis
Complaint Driven Training Data Debugging for Machine Learning Workflows
| Τίτλος: | Complaint Driven Training Data Debugging for Machine Learning Workflows |
|---|---|
| Συγγραφείς: | Flokas, Lampros |
| Έτος έκδοσης: | 2023 |
| Συλλογή: | Columbia University: Academic Commons |
| Θεματικοί όροι: | Computer science, Machine learning, Debugging in computer science--Computer programs |
| Περιγραφή: | As the need for machine learning (ML) increases rapidly across all industry sectors, so has theinterest in building ML platforms that manage and automate parts of the ML life-cycle. This has enabled companies to use ML inference as a part of their downstream analytics or their applications. Unfortunately, debugging unexpected outcomes in the result of these ML workflows remains a necessary but difficult task of the ML life-cycle. The challenge of debugging ML workflows is that it requires reasoning about the correctness of the workflow logic, the datasets used for inference and training, the models, and interactions between them. Even if the workflow logic is correct, errors in the data used across the ML workflow can still lead to wrong outcomes. In short, developers are not just debugging the code, but also the data. We advocate in favor of a complaint driven approach towards specifying and debugging data errors in ML workflows. The approach takes as input user specified complaints specified as constraints over the final or intermediate outputs of workflows that use trained ML models. The approach outputs explanations in the form of specific operator(s) or data subsets, and how they may be changed to address the constraint violations. In this thesis we make the first steps towards our complaint driven approach to data debugging. As a stepping stone, we focus our attention on complaints specified on top of relational workflows that use ML model inference and whose errors are caused by errors in ML model’s training data. To the best of our knowledge, we contribute the first debugging system for this task, which we call Rain. In response to a user complaint, Rain ranks the ML model’s training examples based on their ability to address the user’s complaint if they were removed. Our experiments show that users can use Rain to debug training data errors by specifying complaints over aggregations of model predictions without having to specify the correct label for each individual prediction. Unfortunately, Rain’s ... |
| Τύπος εγγράφου: | thesis |
| Γλώσσα: | English |
| DOI: | 10.7916/1r8f-f405 |
| Διαθεσιμότητα: | https://doi.org/10.7916/1r8f-f405 |
| Αριθμός Καταχώρησης: | edsbas.7F75AA0C |
| Βάση Δεδομένων: | BASE |
| FullText | Text: Availability: 0 CustomLinks: – Url: https://doi.org/10.7916/1r8f-f405# Name: EDS - BASE (ns324271) Category: fullText Text: View record from BASE |
|---|---|
| Header | DbId: edsbas DbLabel: BASE An: edsbas.7F75AA0C RelevancyScore: 851 AccessLevel: 3 PubType: Dissertation/ Thesis PubTypeId: dissertation PreciseRelevancyScore: 851.480346679688 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Complaint Driven Training Data Debugging for Machine Learning Workflows – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Flokas%2C+Lampros%22">Flokas, Lampros</searchLink> – Name: DatePubCY Label: Publication Year Group: Date Data: 2023 – Name: Subset Label: Collection Group: HoldingsInfo Data: Columbia University: Academic Commons – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Computer+science%22">Computer science</searchLink><br /><searchLink fieldCode="DE" term="%22Machine+learning%22">Machine learning</searchLink><br /><searchLink fieldCode="DE" term="%22Debugging+in+computer+science--Computer+programs%22">Debugging in computer science--Computer programs</searchLink> – Name: Abstract Label: Description Group: Ab Data: As the need for machine learning (ML) increases rapidly across all industry sectors, so has theinterest in building ML platforms that manage and automate parts of the ML life-cycle. This has enabled companies to use ML inference as a part of their downstream analytics or their applications. Unfortunately, debugging unexpected outcomes in the result of these ML workflows remains a necessary but difficult task of the ML life-cycle. The challenge of debugging ML workflows is that it requires reasoning about the correctness of the workflow logic, the datasets used for inference and training, the models, and interactions between them. Even if the workflow logic is correct, errors in the data used across the ML workflow can still lead to wrong outcomes. In short, developers are not just debugging the code, but also the data. We advocate in favor of a complaint driven approach towards specifying and debugging data errors in ML workflows. The approach takes as input user specified complaints specified as constraints over the final or intermediate outputs of workflows that use trained ML models. The approach outputs explanations in the form of specific operator(s) or data subsets, and how they may be changed to address the constraint violations. In this thesis we make the first steps towards our complaint driven approach to data debugging. As a stepping stone, we focus our attention on complaints specified on top of relational workflows that use ML model inference and whose errors are caused by errors in ML model’s training data. To the best of our knowledge, we contribute the first debugging system for this task, which we call Rain. In response to a user complaint, Rain ranks the ML model’s training examples based on their ability to address the user’s complaint if they were removed. Our experiments show that users can use Rain to debug training data errors by specifying complaints over aggregations of model predictions without having to specify the correct label for each individual prediction. Unfortunately, Rain’s ... – Name: TypeDocument Label: Document Type Group: TypDoc Data: thesis – Name: Language Label: Language Group: Lang Data: English – Name: DOI Label: DOI Group: ID Data: 10.7916/1r8f-f405 – Name: URL Label: Availability Group: URL Data: https://doi.org/10.7916/1r8f-f405 – Name: AN Label: Accession Number Group: ID Data: edsbas.7F75AA0C |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edsbas&AN=edsbas.7F75AA0C |
| RecordInfo | BibRecord: BibEntity: Identifiers: – Type: doi Value: 10.7916/1r8f-f405 Languages: – Text: English Subjects: – SubjectFull: Computer science Type: general – SubjectFull: Machine learning Type: general – SubjectFull: Debugging in computer science--Computer programs Type: general Titles: – TitleFull: Complaint Driven Training Data Debugging for Machine Learning Workflows Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Flokas, Lampros IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Type: published Y: 2023 Identifiers: – Type: issn-locals Value: edsbas – Type: issn-locals Value: edsbas.oa |
| ResultId | 1 |