Report for Scalable I/O, July 1995

Contact: EOSDIS Project. We have also developed a suite of programs for our NSF/ARPA Grand Challenge project on Land Cover Dynamics as well as an out-of-core sparse factorization code. The Grand Challenge programs are undergoing a process of optimization and refinement. The applications effort is also supported by funds from EOSDIS and the Land Cover Dynamics NSF/ARPA Grand Challenge project.

Library Development:

The University of Maryland has developed two different I/O libraries. The first library is a two-phase I/O library oriented towards loosely synchronous programs whereas the second library is simpler and is designed to handle coarse-grain independent requests. The two-phase I/O library was completed approximately 8 months ago; this library has been extensively tested on the SP-2.

As we analyzed the EOSDIS sensor data analysis programs and developed our own programs for land cover dynamics, we came to the conclusion that two-phase I/O was not necessarily the best approach to optimizing parallel I/O performance. This prompted us to develop our second library which appears to be better suited to this class of problems. We have implemented this library on the 16 processor IBM SP-2 at the University of Maryland. Multi-threading allows both the application and the library to use the high performance switch on the SP-2 in high bandwidth user-space mode. In joint work between Maryland and Argonne, we are making use of both Maryland I/O libraries, along with Argonne's Nexus software to find the most effective strategies for optimizing parallel I/O performance for programs that exhibit various classes of I/O behavior.

Compilers:

We have developed an Interprocedural Partial Redundancy Elimination (IPRE) algorithm that will be used to carry out optimized placement of asynchronous I/O routines. The algorithm will be presented in a paper that will be presented at Supercomputing '95.

1996 Plans:

Libraries:

We plan to carry out a comprehensive performance study of the effectiveness of various parallel I/O optimizations on a broad range of application codes. This study will lead to a the Compiler and Runtime group agreeing on a parallel I/O library API, and to reference I/O library implementations.

Compilers:

The Interprocedural Partial Redundancy Elimination (IPRE) algorithm has been successfully implemented as part of the Fortran D System framework but IPRE has not yet been used for parallel I/O optimization (IPRE has been used to place collective communication calls). We will use the Interprocedural Partial Redundancy Elimination (IPRE) algorithm to carry out optimized placement of asynchronous I/O calls.

Applications:

At Supercomputing '95, we will use the IWAY to demonstrate our suite of sensor data query, analysis and image processing modules. The sensor database, the database query infrastructure and the image processing routines will reside on our SP-2 at the University of Maryland. We will develop a suite of sensor data analysis based parallel I/O templates; these templates will be made available to the parallel I/O consortium and to the community at large.

Technology Transfer:

We are working closely with EOSDIS; we are using EOSDIS applications to motivate our scalable I/O work and we are making use of our Scalable I/O libraries in porting and benchmarking EOSDIS codes.