Academic Journal
Revolutionizing Financial Data Processing with Cloud-Native Pipelines
| Title: | Revolutionizing Financial Data Processing with Cloud-Native Pipelines |
|---|---|
| Authors: | Sahini Dyapa |
| Source: | Journal of Computer Science and Technology Studies; Vol. 7 No. 8; 131-141 ; 2709-104X |
| Publisher Information: | Al-Kindi Center for Research and Development |
| Publication Year: | 2025 |
| Subject Terms: | Cloud-native data pipelines, Financial data processing, ETL transformation, Data architecture modernization, Real-time analytics |
| Description: | This article explores the transformative impact of cloud-native data pipelines on financial data processing, addressing the limitations of traditional batch processing methodologies. It examines how financial institutions have historically struggled with processing delays, limited scalability, inflexible architectures, inefficient resource utilization, and challenges to data consistency. The article outlines a comprehensive migration strategy that leverages Databricks, Snowflake, and Apache Airflow to create a modern, modular architecture with distinct layers for data ingestion, processing, storage, orchestration, and delivery. The implementation achieves substantial improvements in processing speed, scalability, cost efficiency, reporting timeliness, and analytical capabilities. The article provides a detailed examination of the architectural components, implementation considerations, and measurable outcomes of this technological transformation. It concludes by outlining future enhancement opportunities, including machine learning integration, real-time streaming capabilities, enhanced governance frameworks, and expansion to additional data domains. Throughout, the article emphasizes how cloud-native architectures enable financial institutions to maintain a competitive advantage in an increasingly data-driven landscape through improved decision-making capabilities and operational efficiency. |
| Document Type: | article in journal/newspaper |
| File Description: | application/pdf |
| Language: | English |
| Relation: | https://al-kindipublisher.com/index.php/jcsts/article/view/10462/9189 |
| DOI: | 10.32996/jcsts.2025.7.8.16 |
| Availability: | https://al-kindipublisher.com/index.php/jcsts/article/view/10462 https://doi.org/10.32996/jcsts.2025.7.8.16 |
| Rights: | https://creativecommons.org/licenses/by/4.0/ |
| Accession Number: | edsbas.F55AEE6C |
| Database: | BASE |
| DOI: | 10.32996/jcsts.2025.7.8.16 |
|---|