Academic Journal
Self-Healing AI-Native Real-Time Data Pipelines: Autonomous Resilience For Large-Scale Streaming Systems.
| Τίτλος: | Self-Healing AI-Native Real-Time Data Pipelines: Autonomous Resilience For Large-Scale Streaming Systems. |
|---|---|
| Συγγραφείς: | Rajamani, Yogesh Pugazhendhi Duraisamy |
| Πηγή: | Journal of International Crisis & Risk Communication Research (JICRCR); 2026, Vol. 9 Issue 1, p440-449, 10p |
| Θεματικοί όροι: | Data pipelining, Streaming technology, Real-time computing, Automatic control systems, Anomaly detection (Computer security), Failure analysis |
| Περίληψη: | In large streaming platforms today, there are common operational issues, such as data drift, throughput degradation, partition imbalance, and cascading failures, that impact availability and performance. Existing monitoring and rule-based automatic remediation solutions are unsuitable for workloads with millisecond-level latency and high availability needs. This article introduces a fully self-healing AI-native real-time data pipeline that integrates machine learning into the control plane of the streaming platform. It presents an end-to-end architecture that leverages graph neural networks and transformers for hybrid anomaly detection, LSTM-based predictive fault modeling, and reinforcement learning-based agents that autonomously select the best remediation policy (e.g., dynamic resource scaling, partition rebalancing, and dataflow rerouting). The framework implements continuous healing based on the detect-diagnose-predict-decide-act-verify-learn loop. Evaluating the framework with synthetic and real-world high-throughput streaming workloads shows improvements in downtime, latency, fault domains, and resource utilization to establish a new model of autonomous stream processing infrastructures that can continue to operate mission-critical workloads in cloud, hybrid, and edge environments. [ABSTRACT FROM AUTHOR] |
| Copyright of Journal of International Crisis & Risk Communication Research (JICRCR) is the property of Journal of International Crisis & Risk Communication Research and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.) | |
| Βάση Δεδομένων: | Complementary Index |
| FullText | Text: Availability: 0 |
|---|---|
| Header | DbId: edb DbLabel: Complementary Index An: 194238772 RelevancyScore: 1041 AccessLevel: 6 PubType: Academic Journal PubTypeId: academicJournal PreciseRelevancyScore: 1041.06604003906 |
| IllustrationInfo | |
| Items | – Name: Title Label: Title Group: Ti Data: Self-Healing AI-Native Real-Time Data Pipelines: Autonomous Resilience For Large-Scale Streaming Systems. – Name: Author Label: Authors Group: Au Data: <searchLink fieldCode="AR" term="%22Rajamani%2C+Yogesh+Pugazhendhi+Duraisamy%22">Rajamani, Yogesh Pugazhendhi Duraisamy</searchLink> – Name: TitleSource Label: Source Group: Src Data: Journal of International Crisis & Risk Communication Research (JICRCR); 2026, Vol. 9 Issue 1, p440-449, 10p – Name: Subject Label: Subject Terms Group: Su Data: <searchLink fieldCode="DE" term="%22Data+pipelining%22">Data pipelining</searchLink><br /><searchLink fieldCode="DE" term="%22Streaming+technology%22">Streaming technology</searchLink><br /><searchLink fieldCode="DE" term="%22Real-time+computing%22">Real-time computing</searchLink><br /><searchLink fieldCode="DE" term="%22Automatic+control+systems%22">Automatic control systems</searchLink><br /><searchLink fieldCode="DE" term="%22Anomaly+detection+%28Computer+security%29%22">Anomaly detection (Computer security)</searchLink><br /><searchLink fieldCode="DE" term="%22Failure+analysis%22">Failure analysis</searchLink> – Name: Abstract Label: Abstract Group: Ab Data: In large streaming platforms today, there are common operational issues, such as data drift, throughput degradation, partition imbalance, and cascading failures, that impact availability and performance. Existing monitoring and rule-based automatic remediation solutions are unsuitable for workloads with millisecond-level latency and high availability needs. This article introduces a fully self-healing AI-native real-time data pipeline that integrates machine learning into the control plane of the streaming platform. It presents an end-to-end architecture that leverages graph neural networks and transformers for hybrid anomaly detection, LSTM-based predictive fault modeling, and reinforcement learning-based agents that autonomously select the best remediation policy (e.g., dynamic resource scaling, partition rebalancing, and dataflow rerouting). The framework implements continuous healing based on the detect-diagnose-predict-decide-act-verify-learn loop. Evaluating the framework with synthetic and real-world high-throughput streaming workloads shows improvements in downtime, latency, fault domains, and resource utilization to establish a new model of autonomous stream processing infrastructures that can continue to operate mission-critical workloads in cloud, hybrid, and edge environments. [ABSTRACT FROM AUTHOR] – Name: Abstract Label: Group: Ab Data: <i>Copyright of Journal of International Crisis & Risk Communication Research (JICRCR) is the property of Journal of International Crisis & Risk Communication Research and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract.</i> (Copyright applies to all Abstracts.) |
| PLink | https://search.ebscohost.com/login.aspx?direct=true&site=eds-live&db=edb&AN=194238772 |
| RecordInfo | BibRecord: BibEntity: Languages: – Code: eng Text: English PhysicalDescription: Pagination: PageCount: 10 StartPage: 440 Subjects: – SubjectFull: Data pipelining Type: general – SubjectFull: Streaming technology Type: general – SubjectFull: Real-time computing Type: general – SubjectFull: Automatic control systems Type: general – SubjectFull: Anomaly detection (Computer security) Type: general – SubjectFull: Failure analysis Type: general Titles: – TitleFull: Self-Healing AI-Native Real-Time Data Pipelines: Autonomous Resilience For Large-Scale Streaming Systems. Type: main BibRelationships: HasContributorRelationships: – PersonEntity: Name: NameFull: Rajamani, Yogesh Pugazhendhi Duraisamy IsPartOfRelationships: – BibEntity: Dates: – D: 01 M: 01 Text: 2026 Type: published Y: 2026 Identifiers: – Type: issn-print Value: 25760017 Numbering: – Type: volume Value: 9 – Type: issue Value: 1 Titles: – TitleFull: Journal of International Crisis & Risk Communication Research (JICRCR) Type: main |
| ResultId | 1 |