The International Conference for High Performance Computing, Networking, Storage, and Analysis
The International Conference for High Performance Computing, Networking, Storage, and Analysis
Workshop: ISAV 2022: In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization
Authors: Mathew Boyer and Wesley Brewer (HPC Modernization Program (HPCMP) User Productivity Enhancement and Training (PET)), Dylan Jude (US Army Combat Capabilities Development Command Aviation & Missile Center), and Ian Dettwiller (US Army Engineer Research and Development Center (ERDC))
Abstract: Machine learning (ML) has become ubiquitous within the sciences due to its ability to perform a wide array of tasks which add value within traditional workflows. These models can provide advanced data analytics through dimensionality reduction, pattern recognition, and clustering. For large-scale simulations, post hoc data analysis requires writing and reading large quantities of data, which can severely limit the rate. In situ analysis can reduce the frequency and quantity of data written to disk but requires the integration of simulations with ML methods, which poses a software development challenge. In this talk, we will present an approach to integrating simulations with ML models through an inference server and remote procedure calls (RPCs). By separating the machine learning into one or more independent processes, the inference calls can be made within drop-in functions using RPCs with minimal modifications to the existing code and can be scaled across parallel processes with MPI. While the deep learning platform, TensorFlow, is typically considered a Python tool, RPCs can couple a TensorFlow model server with applications written in a wide variety of languages. We will demonstrate the computational efficiency and scalability of the approach across a series of use cases, such as deploying machine-learned surrogate models in simulations and enabling ML super-resolution in visualization tools.