The Board of Trustees of the University of Illinois 801 South Wright Street Urbana, Illinois 61801
Andrew A. Chien
Department of Computer Science
University of Illinois
1304 West Springfield Avenue
Urbana, Illinois 61801
Phone: 217-333-6844
Fax: 217-244-6500
Email: achien@cs.uiuc.edu
Level of Participation 30%
http://www-csag.cs.uiuc.edu/projects/hpvm.html
High Performance Virtual Machines (HPVMs) can increase the accessibility and delivered performance of distributed computational resources for high performance computing applications. Successful HPVM's will reduce the effort required to build efficient parallel applications on distributed resources, increase the performance delivered to those applications, and leverage parallel software tools from existing parallel systems to distributed environments.
The rapidly increasing performance of low-cost computing systems has produced a rich environment for desktop, distributed, and wide-area computing. However this wealth of computational resources has not been effectively harnessed for high performance computing. High Performance Virtual Machines (HPVMs) are a new technology which leverage the software tools and developed understanding of parallel computation on scalable parallel systems to exploit distributed computing resources. The objective is to reduce the effort to build high performance applications on distributed systems.
High Performance Virtual Machines depend on building a uniform, portable abstraction -- a virtual machine -- with predictable, high performance characteristics. To successfully insulate application programs, a virtual machine must (1) deliver a large fraction of the underlying hardware performance, (2) virtualize resources to provide portability and to reduce the effort in building application programs, and (3) deliver predictable, high performance. The project is developing novel technology that leverages commodity components (hardware and software) to deliver high performance communication over cluster and wide area interconnects, predictable communication and computation in a dynamic resource environment, and uniform access to resources (e.g. files, mass storage, embedded sensors). The HPVM project involves not only the development of novel communication, scheduling, and resource management technologies, but also dissemination of a series of software releases which embody these ideas.
Demonstrated the largest parallel Windows NT cluster (192 processors, 96 2-way multiprocessor nodes), using HPVM software technology for cluster management, scheduling, and communication. This effort not only demonstrated the viability of Windows NT platforms for high performance computing, it also drove rapid improvement in a number of commercial software systems (Platform's LSF and PGI's HPF compiler) as well as HPVM. In addition, the visibility of the effort has catalyzed a significant fraction of the high performance computing community to adopt Windows NT more rapidly, improving the breadth and quality of application software for high performance computing available on Windows NT. This demonstration was achieved by a joint effort with NCSA, the National Computational Science Alliance.
Developed and distributed an improved high performance messaging layer and interface, Fast Messages 2.1, which delivers peak network hardware 92 megabytes per second and 8 microseconds latency) to both parallel computing (MPI, Shmem Put/Get, and Global Arrays) and distributed computing API's (RPC). The major improvement in functionality for FM2.1 is supporting multiprocess use and dynamic formation of network groups. This software operates on both Windows NT and Linux.
Developed a predictable communication layer called FM-QoS, based on Fast Messages, which uses network-level feedback to synchronize network nodes, and deliver network quality of service without special hardware support in the network switches. This layer enables QoS to be assured without increased network cost or slower hardware due to increased switching complexity. The current prototypes achieve 40 microsecond allocation granularity and link bandwidths of 400 megabits per second with no more than a few microseconds jitter. (Windows NT)
Developed an implementation architecture and design for Fast Messages on Compaq's Servernet cluster interconnect. This implementation design includes several innovations which overcome significant shortcomings in the Servernet design including a lack of support for network virtualization and remote synchronization. We are confident this implementation will deliver the full peak Servernet performance with comparable low overhead to that of our Myrinet Fast Messages implementation. (Windows NT)
Interfaced the HPVM Shmem Put/Get layer to the Portland Group's commercial High Performance Fortran (HPF) compiler which enables programming of HPVM clusters with a single address space, high level programming model. This integration exploits the mature commercial compiler technology from the Portland Group, and we expect it to become as popular a programming vehicle for HPVM clusters as the MPI interface. (Windows NT)
Improved the scalability of Platform Computing's Load Sharing Facility (LSF) product by deeply integrating it with HPVM and driving their requirements, isolating scalability problems, and proposing and contributing fixes, improving software scalability from four nodes to over 100 nodes in our large-scale Windows NT cluster demonstration.
Designed and developed implementations and demonstrated coordinated scheduling (coscheduling) based on loadable kernel mode drivers in Windows NT. These drivers use network communication traffic to influence the thread scheduling choices made by the NT kernel scheduler. Not only did this effort demonstrate the viability of the approach for providing predictable parallel computational performance, the experience also has produced clear requirements for the scheduling interfaces needed to make the NT kernel scheduler easily externally customizable.
Designed and developed prototype high performance remote procedure call invocation mechanisms exploiting the low-overhead and low latency communication mechanisms of the Fast Messages layer (and other emerging user-level network interfaces). These implementations of Java Remote Method Invocation (RMI) and Microsoft RPC (MSRPC) achieve performance of 47 microseconds and 105 microseconds for round trip null RPC's respectively. These times reflect performance improvements of 3 to 8x, bringing distributed RPC times to close to that for local RPC's.
The major objectives for the fiscal 1998 year for the High Performance Virtual Machines are:
In addition to ongoing technology transfer through software releases, this year we have achieved two major additional technology transitions in FY1998.
The HPVM project has produced numerous software releases of Fast Messages 2.0x, MPI-FM, and related software. These packages are widely disseminated and used. We now have approximately 50 source code licensees (typically researchers) and have logged over 1000 downloads from over 500 distinct users. More germane are the approximately 40 source licensees and over 200 users since the HPVM 1.0 release in August 1997. These users come from Department of Defense laboratories, NASA, the U.S. national laboratories, as well as academic and corporate researchers from around the world. In numerous sites, FM-related software releases are being used for production computing. Fast Messages technology is being used in a range of DARPA, NSF, and NASA research projects too numerous to list. The details of available software elements are included below.
Prepared July 1998
webmaster