Treffer: dispel4py: A Python framework for data-intensive scientific computing.

This paper presents <monospace>dispel4py</monospace>, a new Python framework for describing abstract stream-based workflows for distributed data-intensive applications. These combine the familiarity of Python programming with the scalability of workflows. Data streaming is used to gain performance, rapid prototyping and applicability to live observations. <monospace>dispel4py</monospace> enables scientists to focus on their scientific goals, avoiding distracting details and retaining flexibility over the computing infrastructure they use. The implementation, therefore, has to map <monospace>dispel4py</monospace> abstract workflows optimally onto target platforms chosen dynamically. We present four <monospace>dispel4py</monospace> mappings: Apache Storm, message-passing interface (MPI), multi-threading and sequential, showing two major benefits: a) smooth transitions from local development on a laptop to scalable execution for production work, and b) scalable enactment on significantly different distributed computing infrastructures. Three application domains are reported and measurements on multiple infrastructures show the optimisations achieved; they have provided demanding real applications and helped us develop effective training. The <monospace>dispel4py.org</monospace> is an open-source project to which we invite participation. The effective mapping of <monospace>dispel4py</monospace> onto multiple target infrastructures demonstrates exploitation of data-intensive and high-performance computing (HPC) architectures and consistent scalability. [ABSTRACT FROM AUTHOR]

Copyright of International Journal of High Performance Computing Applications is the property of Sage Publications Inc. and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)