The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: ISAV 2022: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization
Authors: Cyrus Harrison (Lawrence Livermore National Laboratory); Matthew Larsen (Luminary Cloud Inc); and Brian S. Ryujin, Adam Kunen, Arlie Capps, and Justin Privitera (Lawrence Livermore National Laboratory)
Abstract: Data representation and coupling between scientific libraries is a key challenge to building a vibrant ecosystem of HPC simulation tools. From bespoke data structures to hundreds of file-based data models, the myriad of possible choices involved both enables key features and blocks adoption of others. Connecting data between code bases requires agreeing on or adapting between data representations. While in some cases this process is trivial, for more complicated cases, adapting data becomes a costly barrier. Conduit was designed within this context to help meet the key challenge of sharing data across HPC simulation tools by providing a dynamic API to describe in-memory data. It supports coupling simulations and connecting simulations to analysis and I/O libraries. This paper provides a broad overview of Conduit, background on the evolution of the project, and details on recently added features relevant to in situ use cases.