HPVM 1.9 User Documentation

Go to the first, previous, next, last section, table of contents.

HPVM Components

HPVM is a layered system where the fundamental design allows the underlying hardware performance to be exposed through the protocols to the user's application.

Logically the system is split into three components:

The development environment and link libraries. These include the FM, MPI, SHMEM, BSP, and GA link libraries for building user applications.
The Fast Messages (FM) runtime. This includes the required components to run compiled programs on the actual cluster.
(Optional) The LSF setup and HPVM Job Submission Client. This is for installations that want to use the remote job submission client with LSF.

The development environment contains example programs, include files, sample make files, prebuilt libraries, and documentation. You should install this environment where users usually compile and link their applications. This environment is installed through the Development Libraries install option and should be placed on the machine where users usually develop, compile, and link their applications. It is unnecessary to place these files on the compute nodes of the cluster.

All programs that run on the compute cluster use FM at the lowest level. MPI-FM, for example is layered on top of FM and programs would link with both FM.lib and MPI.lib.

Important! Compiling user programs requires either Microsoft Visual C++ 5.0/6.0 and/or Digital Fortran. The example makefiles assume that the environment variables for these compilers are properly setup. The file `DevStudio/VC/bin/vcvars32.bat' will setup the environment correctly for Visual C++, and `DevStudio/DF/bin/dfvars.bat' will setup the environment for Digital Fortran. These batch files are provided by the compiler setup program. Also, the example files assume the use of Microsoft's nmake.

Workstation-only Setup

New for HPVM1.9! A Workstation Setup is now available that allows users to develop, debug, and test an FM program on a local workstation without the need of high-performance networking cards. This single-node setup uses a new shared-memory transport to provide interprocess communication. The setup automatically installs all of the needed services to run an FM job on your local workstation. Executables created in this environment can be run on an HPVM 1.9 cluster without recompiling or relinking.

FM Runtime

Since all of the supplied APIs (MPI, SHMEM, GA, BSP) use FM either directly or indirectly, it is only the FM runtime that must be installed on the cluster. The FM runtime is split into three pieces:

Compute Node consisting of the high-speed interconnect (with supporting hardware and software), Context Manager, FM dynamically-linked library (FM.DLL), and the VI Provider dynamically-linked library (VIPL.DLL).
High-Speed Interconnect HPVM1.9 Supports two different high-speed networks: Myrinet(12) and Giganet(VIA)(13). The VIA transport used by FM has only been tested on Giganet GNN1000 interfaces. In addition, a custom device driver, NIC Lanai Control Program, and network configuration file are provided for those desiring Myrinet support.
Global Manager only one machine should be designated for the global manager. The global manager keeps a small amount of essential state so that application programs can coordinate at startup. The global manager need not run on node that has a high-speed network.

Configuring a Compute Node

Each node in the cluster must have two basic FM components installed: A High-Speed interconnect and the Context Manager (fmcm.exe). In addition, if a Myrinet cluster is being built, the HPVM-supplied device driver, network card control program (fmmcp.dat), and the network configuration file (fmconfig.txt), must also be present.

You should select the Compute Node setup type for each machine in your cluster. Aside from the usual NT-style setup questions, you must be prepared to answer a few questions at setup.

DNS Name of Global Manager - input the network name of the machine where you will install the global manager. All nodes put the same name here.
High-Speed Network - Choose either, both, or none of the following: (Choosing none is what the Workstation Setup does automatically)
1. Myrinet - You have a myrinet device in each node
2. Giganet(VIA) - You have VIA interface (Only the Giganet GNN1000 has been tested).
Default Transport - If select both high-speed interconnects, you need to specify which one is the default transport. A registry key (See below) can be modified to change which transport FM programs use.
Cluster Dimensions - FM requires some configuration information to indicate the maximum dimensions of your cluster. There are "sensible" defaults provided by the setup program, but you should modify as appropriate. You are only asked this question if Myrinet and/or Giganet was selected as high-speed interconnects.
1. # Nodes - (Default: 16) Input the number of compute nodes in your cluster
2. # Contexts - (Default: 2) This is the number of processes/node that may simultaneously access the high-performance network. The maximum number of tasks is therefore given by (# Contexts) X (# Nodes).
3. # Credits - (Default: 8) FM uses credit-based flow control and this number corresponds to the initial number of packets a sender my transmit before requiring a refill from the receiver. Increasing the number of credits boosts point-to-point bandwidth, but requires a larger amount of pinned (non-paged) memory to be allocated by the driver. (# credits > 1).
A rough estimate of required pinned memory is given as (# Contexts) X [(# Nodes) x (# Contexts) x (# Credits) x (2080 bytes)] Note that the device driver will try to pin down (# Contexts) memory regions. So the default setup requires just over 1MB of pinned memory in two 0.5MB regions. We have successfully pinned four 28.9 MB regions (approximately 115MB) on a dual Pentium II machine with 384MB of core memory running NT Server. Please see http://ftp.sunet.se/ftp/pub/NT/mirror-microsoft/kb/Q126/4/02.TXT for a more complete discussion of pinned memory limitations on NT. If you stop the device driver and increase its memory requirements, it may fail to load. Sometimes, simply rebooting the machine will allow the driver to load properly because memory is less fragmented and the driver has a better chance of pinning a larger memory region. If the device driver fails to load because of insufficient resources, you will have to modify the appropriate registry variables to reduce memory consumption.

Important! Once you have completed the Compute Node Setup and have chosen Myrinet, you will need to define a network configuration file. The default file is in `\hpvm1.9\bin\fmconfig.txt'. A later section gives complete details one how to write a network configuration file. Setup will remind you to edit your own network configuration file. If you reboot the machine without making these changes, the context manager will probably not start due to the fact that your machine name is not listed in the fmconfig.txt file. In this case, edit the configuration file appropriately and use the Services applet in the NT control panel to restart the service.

Global Manager Setup

You need to select a single machine to run the global manager service. The global manager only uses TCP/IP so it is not required that it be put on a machine with the high-performance network.

(Optional) LSF setup and HPVM interface

One should note that the FM runtime does not specify how jobs get launched onto individual cluster nodes. Furthermore, the FM runtime does not have any built-in support for job management. Instead, some other mechanism must be employed to start tasks on the individual nodes. Once, tasks are started, the global manager is used to enable initial synchronization of tasks.

This version of HPVM only supports Platform Computing's Load Sharing Facility (LSF) to launch parallel jobs onto cluster nodes. LSF provides a command-line interface to launch jobs as well as a GUI if you are locally logged onto one of the cluster machines or the LSF server. HPVM provides a front-end client (written in Java) to provide a GUI interface to some of the LSF services for remote users.

The client, when connected to the supplied server, allows users to submit batch jobs and monitor some aspects of the cluster such as compute load.

If you desire the LSF support, select the Java Server setup type. Setup asks only one extra question in this case

Select your wwwroot directory (default: \InetPub\wwwroot). Setup will install the java class archive for the client and a sample HTML file called HPVM_Client.html.

Important! This setup also installs the "HPVM Job Submission" service and starts it. The service should run as the LSF Administrator. You should go to the NT Services Control Panel Applet and select the proper username and password for your system (set up will remind you to this).