Academic Journal
Scientific Big Data Visualization: a Coupled Tools Approach
| Τίτλος: | Scientific Big Data Visualization: a Coupled Tools Approach |
|---|---|
| Συγγραφείς: | Artigues, Antoni, Cucchietti, Fernando, Tripiana, Carlos, Vicente, David, Calmet, Hadrien, Marín, Guillermo, Houzeaux, Guillaume, Vázquez, Mariano |
| Συνεισφορές: | Barcelona Supercomputing Center |
| Στοιχεία εκδότη: | South Ural State University (Chelyabinsk, Russia) |
| Έτος έκδοσης: | 2015 |
| Συλλογή: | Universitat Politècnica de Catalunya, BarcelonaTech: UPCommons - Global access to UPC knowledge |
| Θεματικοί όροι: | Àrees temàtiques de la UPC::Enginyeria electrònica, Large scale systems, Database management Computer programs, Big data, Parallel visualisation system, Hadoop file system, Paraview-native VTK, Macrodades, Supercomputadors |
| Περιγραφή: | We designed and implemented a parallel visualisation system for the analysis of large scale time-dependent particle type data. The particular challenge we address is how to analyse a high performance computation style dataset when a visual representation of the full set is not possible or useful, and one is only interested in finding and inspecting smaller subsets that fulfil certain complex criteria. We used Paraview as the user interface, which is a familiar tool for many HPC users, runs in parallel, and can be conveniently extended. We distributed the data in a supercomputing environment using the Hadoop file system. On top of it, we run Hive or Impala, and implemented a connection between Paraview and them that allows us to launch programmable SQL queries in the database directly from within Paraview. The queries return a Paraview-native VTK object that fits directly into the Paraview pipeline. We find good scalability and response times. In the typical supercomputer environment (like the one we used for implementation) the queue and management system make it difficult to keep local data in between sessions, which imposes a bottleneck in the data loading stage. This makes our system most useful when permanently installed on a dedicated cluster. ; This work was partially supported by the Spanish Severo Ochoa Program, grant SEV-2011-00067. We acknowledge PRACE for awarding us access to the resource FERMI based in Italy at CINECA. ; Peer Reviewed ; Postprint (published version) |
| Τύπος εγγράφου: | article in journal/newspaper |
| Περιγραφή αρχείου: | 15 p.; application/pdf |
| Γλώσσα: | English |
| Relation: | http://superfri.org/superfri/article/view/29; info:eu-repo/grantAgreement/MINECO/SEV-2011-00067; info:eu-repo/grantAgreement/EC/FP7/285380/EU/The Productive Robot Apprentice/PRACE; https://hdl.handle.net/2117/84772 |
| DOI: | 10.14529/jsfi140301 |
| Διαθεσιμότητα: | https://hdl.handle.net/2117/84772 https://doi.org/10.14529/jsfi140301 |
| Rights: | Open Access |
| Αριθμός Καταχώρησης: | edsbas.3E199B5D |
| Βάση Δεδομένων: | BASE |
| DOI: | 10.14529/jsfi140301 |
|---|