Preface
The MicroGrid is
a tool which provides the ability to emulate virtual grid infrastructures, enabling
scientific study of grid resource management issues. The MicroGrid emulation tool enables
repeatable, controllable experimentation with dynamic resource management
techniques, a critical activity in the investigation of flexible resource
sharing environments posited for future Global Grid scenarios. MicroGrid experimentation complements
experimentation with physical resources by providing the capability to emulate
both resources not yet deployed (what if scenarios) and rare events such as
catastrophic failures. As such, the
MicroGrid supports scientific evaluation of adaptive software, resource
management algorithms and implementations, and general resource and
applications control algorithms over a wide variety of Grid configurations.
For more information,
the interested reader is referred to:
Validating and
Scaling the MicroGrid: A Scientific Instrument for Grid Dynamics, Xin Liu, Huaxia Xia, and Andrew Chien, to be appeared in Journal
of Grid Computing.
The MicroGrid
offers many advantages for grid researchers, such as a tunable process
scheduler, pervasive instrumentation for system profiling, and scalability from
local to wide area networks.
Assumptions
Readers should be
proficient in the use of a standard UNIX shell, gzip, tar and the build tool
GNU make. In addition, general
familiarity with the principles of IP networking is helpful. Finally, if you do not understand how to use
XML, then developing useful configuration files and network topologies might be
difficult, although you should be able to build simple experiments easily.
Font Conventions Used in This Manual
Courier
Font is used for:
Italic Courier Font is used for:
·
Arguments
to MicroGrid functions, since they could be typed in code as shown but are
arbitrary.
Italic Arial is used for:
·
MicroGrid
specific terms.
Boldface is used for:
·
Chapter
and section headings.
2.1 Contents
The MicroGrid package contains:
Download the MicroGrid distribution from the distribution website (http://www-csag.ucsd.edu/projects/grid/download.html) and unpack it using 'gunzip' and 'tar'. This will create a directory called 'mgrid2', where you will find example MicroGrid resource files, Globus patches, configuration files, and a number of scripts to build common applications.
In order to
successfully install and use the MicroGrid, edit the MicroGrid resource file (mgridrc) and declare the following
variables needed by the build scripts and emulation system.
·
$MGRID_ROOT: MicroGrid installation path
·
$PERL5LIB: Path to your installation of Perl5
·
$MPICHP4_PATH: Path to your installed MPICH ch_p4 device
(eg: /usr/x86-local/mpich/ch_p4)
·
$MPICHG2_PATH: Path to your local installation of the MPICH
Globus device (eg: /usr/x86-local/mpich/globus2)
·
$GLOBUS_BUNDLE_DIR: Path where the Globus bundles can be found. These source bundles are needed to build
MicroGrid enabled Globus services.
Now run 'source mgridrc' to load the variables into your environment.
|
If you are a tcsh user, edit and source the mgrid.tcsh file instead of mgridrc. |
2.3 Building the Emulation Libraries
To build the MicroGrid emulation library with Globus support (mgrid_globus), enter the mgrid2 directory and type './build_mgrid_globus'. If your environment settings are correct, gmake will begin building the emulator libraries and binaries; be patient the project may take a while to finish.
This script should build and install the mgrid-patched Globus 2 to $GLOBUS_LOCATION defined in your mgridrc file. Under $GLOBUS_LOCATION, you should find the following mgrid-globus specific files:
bin/mgridrun
sbin/mgrid-gatekeeper
libexec/mgrid-job-manager
lib/libmgrid.a
This MicroGrid enabled Globus installation will not interfere with your original Globus installation (if any), as long as you set the $GLOBUS_LOCATION environment variable correctly before using it. See the official Globus website for information regarding Globus installation (http://www.globus.org).
If you don’t need Globus support, you can build the standalone MicroGrid emulation library (mgrid). Enter the mgrid2 directory and type './build_mgrid'. Again, be patient since the project may take a while to finish.
2.4 User Deployment
Before you may begin using the MicroGrid, it
must first be deployed to a shared NFS/SMB filesystem available to all the
nodes in your system. To deploy the MicroGrid,
first source your mgridrc (mgridrc.tcsh) file, this updates any variables that may
have changed in your mgridrc after
the initial build. Now run the
mgrid_deploy script under your $MGRID_ROOT directory.
This script will deploy your build of the MicroGrid into the directory
specified by the $MG_CONF environment variable. If this variable has not been declared the mgrid_deploy script will create a directory named ‘.mgrid’ in the user’s home
directory. This directory contains a
number of configuration files and other temporary state files the emulator may
need at runtime. These files include an
updated MicroGrid resource file
mgridrc, the
MicroGrid configuration file mg_conf, and a
number of internal temporary directories.
After deployment
each user must edit the mg_conf file (changing each of the variables
below) to a unique set of values. If
they are not unique bizarre interference may occur.
·
NSE_ADMPORT: Emulator administration port, through which MicroGrid enabled applications
send administrative messages to the emulation.
·
MAP_ADMPORT: The MapServer port, through
which the MicroGrid enabled application will add, remove, and query for
virtual/real IP address mappings.
·
MAP_ADMADDR: The IP address of the physical machine, on which the MapServer daemon will run.
·
SCH_PORT: The IP port, through which
the command line job dispatch client ‘schapi’will send job activation, termination, and
control messages to MicroGrid process Schedulers.
If you plan to use Globus with the MicroGrid, you must find the correct values for the following variables and edit them in mg_conf appropriately.
· LDAP_SERVER: IP Address or Hostname of the physical machine serving as LDAP server (GIIS).
· LDAP_PORT: Port used by the GIIS.
· LDAP_PASSWORD: Plaintext of the GIIS password.
· basedn: This optional string contains your Globus Domain Name
|
See: $MASSF_DIR/control/mg_conf.example for a more verbose example. |
3.1 Building a Virtual Network
First, source the mgridrc file in your $MG_CONF directory. Now, create a
directory where you will store your network configuration files and launch your
experiments. For the remainder of this
section we refer to this directory as your Working
Project Directory (WPD). We
recommend that you place your WPD in
a folder on a NFS/SMB shared filesystem, so that all machines participating in
the emulation can get copies of the necessary Virtual Network files. After creating a WPD, change to (cd) this directory and copy the project template
from $MGRID_ROOT/examples/template_mgrid_globus/*
into
your current working directory. If you
are targeting a non-Globus application use the files in $MGRID_ROOT/examples/template_mgrid/*. You should now have the following files in
your WPD:
MYGRID.dml
machine.dml
MYGRID.phost
MYGRID.vhost
Makefile
Now, begin editing the sample MYGRID.dml file and save it to a file named <GRIDNAME>.dml. The <GRIDNAME> you have chosen will be used to identify your network configuration. Make a note of this <GRIDNAME>, as you will use it in the following procedures.
Next, edit the machine.dml file in your current WPD and list the physical machines available to the network emulator. This file contains a sequence of entries, of the following format:
MACHINE
[IPADDR hostname ARCHITECTURE os_type PARTITION
processor_number]
The hostname field specifies the hostname, or IP address, of a physical host that will run the emulator. The os_type, and processor_number fields respectively, specify the current operating system and the number processors available on the target machine.
àDue to an implementation limitation the processor_number must be set to 1.
àBeware: MicroGrid options used with mpirun, do not allow you to include the machine from which you launch the MicroGrid in your machine.dml file.
Next, edit the <GRIDNAME>.phost file and list the physical machines available to the application. For each physical machine, include one line as below:
machine_name CPU_rate CPU_count memory
CPU_rate is a relative CPU speed, you can use MHz, FLOPS, or even the ratio to a “standard” CPU.
àDue to an implementation limitation the CPU_count must be set to 1 (only uniprocessor modeling is
supported); the memory field is not used currently.
Edit the <GRIDNAME>.vhost to define the capabilities of each virtual hosts you are going to use. The first line of the file defines the CPU downgrade, the factor by which the emulation rate is slower than wall clock time. For example:
cpu_downgrade=5
Then, for each virtual host, include one line as below:
virtual_name virtual_ID CPU_rate CPU_count
memory [gatekeeper]
Where virtual_name is the name of the virtual machine used by the user to identify the machine; a “*” can be used if user doesn’t care about the name. The virtual_ID variable is the host ID from topology file. The CPU_rate variable specifies the relative speed of virtual CPU, which should be comparable to the CPU_rate of the physical CPU. The last item, gatekeeper indicates the virtual machines where you want to run gatekeeper.
àDue to an implementation limitation the CPU_count must be set to 1; the memory field is
not used currently.
Now, edit the makefile and set the $GRIDNAME variable to the name of your <GRIDNAME> file and then execute make.
|
The MicroGrid emulator uses a virtual network topology file that complies with the original SSFNET Domain Modeling Language (DML). You can find more information about the SSFNET DML file format here: http://www.ssfnet.org/InternetDocs/ssfnetTutorial-1.html Though MicroGrid DML files follow the same format used by SSFNET, they have been modified to include an emulation_rate setting that specifies the CPU scaling factor. This value indicates the rate by which time progresses in the application; Larger numbers result in a slower than real-time rate of progress. |
If make was able to successfully build your emulated network, you will find the following files in your WPD:
· emulator-LINUX: The emulator executable.
· mpirun-emulator: The script used to launch the emulator.
· <GRIDNAME>.xml: The virtual hosts information to be uploaded to the GIIS server.
· FWTB: A mapping of the virtual IP addresses in the virtual network to the physical host that provides emulation for each virtual host. At present virtual IP addresses are assigned automatically from the network DML file. You should examine this FWTB file to learn how virtual IPs, and DML addresses, have been assigned.
· <GRIDNAME>.mapfile: The physical/virtual mapping information
· <GRIDNAME>-app.xml: The configuration used to run application
3.2 Launching the Emulator
Congratulations, you
are now ready to launch the emulator!
% mpirun-emulator
This starts the emulator process, and you can now run experiments that communicate through the virtual network. There are a variety of parameters for mpirun-emulator; examples of which can be found in $MGRID_ROOT/template_mgrid/run.
4.1 MicroGrid Enabling through static
linker interception
Now prepare your simple application to run with the mgrid by following these three steps.
1.) Modify your application makefile to include defs.mk as follows:
include
${MASSF_DIR}/wrapper/defs.mk
2.) Add ${WRAP_LIST} to the linker options in your makefile
3.) then delete the preexisting application binary, and rebuild using the makefile.
Your application is now MicroGrid enabled.
You can run original binary codes on the MicroGrid by dynamic binding to the MicroGrid library, please refer to subsection 4.7 for instructions.
4.2 Virtual Compute Resources
The forwarding table (FWTB)
and mapping table (<GRIDNAME>.mapfile) provide some useful information
about how the virtual resources are mapped on to the physical resources.
The forwarding table contains
translations from virtual IP addresses in the emulation to DML node names in
the virtual network description (these are not IP addresses). The emulator assigns IP addresses and
configures routing tables appropriately, consistent with CIDR hierarchical
address management. The mgrid system uses the FWTB
translations to inject/extract application messages to/from the network
emulator.
Each line of the FWTB file
has the following format:
IPAddress/Netmask Physical-Node Node-Type DML-Address
For example:
0.0.1.46/30 machine-23 Router 3:3(1)
This entry indicates that the DML
node with address 3:3(1) has been assigned the virtual IP
Address 0.0.1.46 with a subnet mask of 30. This DML node is hosted on the physical
node machine-23, and is functioning as a router.
There are two Node-Types, router and host. Applications and jobs run only on nodes of type host.
The mapping table contains more
detailed information for virtual hosts.
Each line of the mapping table has the following format:
Virtual-name IPAddress Physical-Node
CPU-percentage memory
For example:
n10.caltech.teragrid.org 0.0.2.40
compute-0-5 32 512
This entry indicates that the virtual host n10.caltech.teragrid.org is assigned IP address 0.0.2.40 and will run on physical machine compute-0-5. The virtual host will use at most 32% of available CPU cycles and 512MB memory.
There is another file that is generated: <GRIDNAME>-app.xml. This file defines the mapping information in XML format and is used by the scheduler. It includes a sequence of entry elements called a hostmap.
<hostmap>
<entry>
<v_host>0.0.1.10</v_host>
<p_host>compute-1-0</p_host>
<speed>90</speed>
<memory>512</memory>
</entry>
...
</hostmap>
The hostmap example above specifies that jobs for virtual host 0.0.1.10 will be executed on physical host compute-1-0 and use 90% of the available CPU cycles and 512MB memory on that physical host.
4.3
Configuring Your Application
Having already configured your virtual resource environment, your next step is to write the job description file that specifies where and how each application process will be executed.
An example file, named jobs-cs.xml, can be found in your WPD (a copy of this file can be found in $MGRID_ROOT/examples/template_mgrid). This file is used to specify a sequence of job entities known as a joblist.
<joblist>
<stdout>/home/ksmith/tmp/TcpServer.out</stdout>
<stderr>/home/ksmith/tmp/TcpServer.err</stderr>
<job>
<exec>/home/ksmith/tcptest/TcpServer</exec>
<startime>0</startime>
<param>5234</param>
<v_host>0.0.1.130</v_host>
</job>
<job>
<exec>/home/ksmith/tcptest/TcpClient</exec>
<startime>2</startime>
<param>0.0.1.130
5234</param>
<v_host>0.0.0.34</v_host>
</job>
</joblist>
Each job specifies, an application executable, and the number of seconds after the joblist is submitted that this job should run. Each virtual host is declared using a v_host tag and command line variables may be specified using a param tag.
The example above redirects stdout to /home/ksmith/tmp/TcpServer.out and stderr to /home/ksmith/tmp/TcpServer.err. It then starts a TcpServer application with a single parameter 5234. The application is assigned to v_host 0.0.1.130. Finally a TcpClient application with parameters 0.0.1.130 and 5234, on v_host 0.0.0.34, starting 2 seconds after the joblist is submitted.
You are almost ready to start you application, but first you have to start the service daemons.
4.4 Starting the Service Daemons
To start the
MicroGrid service daemons execute the following command.
% start_daemon.pl <GRIDNAME>
You have now started a MapServer on the local machine and a process Scheduler on each physical resource defined in the resource XML file (see: Section 4.3.2).
4.5 Launching the Application
Congratulations, you can now start
your application with the following command.
% submit_job.pl <GRIDNAME> MYJOB.xml
4.6
Launching the Application Manually
For most users, launching the
application through the submit_job.pl script is the most convenient approach. But
if you need more control (for example, you want to debug), you can launch each
single process manually by following these steps:
1) Login to the machine to execute
this process.
2) Set environment variable $MGHOSTNAME to the IP address of the virtual host for this
process. For example, you can
% export
MGHOSTNAME=0.0.1.20
By setting this environment
variable, you have turned this shell to a console of the virtual hostname 0.0.1.20. All mgrid jobs executed under this shell
will use this virtual IP address as its host IP address.
3) Launch the job just as you normally would.
Remember, the MicroGrid Scheduler
does not control jobs launched in this manner, as a result such experiments may
be somewhat less accurate.
4.7
Run application through dynamic binding to the MicroGrid libary
You can run original binaries (without
relinking to the MicroGrid library) through
dynamic binding. Using this method, you
can run almost any programs, including Java, Python, etc.
First, define the environment
variables “LD_PRELOAD” and “MGHOSTNAME”:
% export
LD_PRELOAD=$MGRID_ROOT/massf/wrapper/libmgrid.so
% export MGHOSTNAME=0.0.1.20
Then you can run your binaries as
usual. For example:
%
/bin/hostname
will print “0.0.1.20” instead of your physical host
name.
ß Remember, you must compile the MicroGrid using gcc 3.2 or higher
version to use this feature.
|
See: $MGRID_ROOT/examples/template_mgrid/ld_exec.sh |
5.1 MicroGrid Enabling Globus
To prepare your
Globus application for use with the MicroGrid, you must perform the following
three steps.
1.) Modify your application makefile to include defs.mk as follows:
include
${MASSF_DIR}/wrapper/defs.mk
2.) Add ${WRAP_LIST} to the linker options in your makefile.
3.) Delete the preexisting application binary, and then rebuild using the makefile.
Your Globus
application is now MicroGrid enabled.
5.2 Virtual Compute Resources
This section discusses three files
that provide useful information about how the virtual resources are mapped to
the physical resources: the forwarding table (FWTB), mapping table (<GRIDNAME>.mapfile), and the application mapping
table (<GRIDNAME>-app.xml).
If you have already read
subsection 4.2 you may skip most of the remainder of this subsection. The only
difference is there’s a ‘<gatekeeper>’ entry in the application-mapping table.
The forwarding table contains
translations from virtual IP addresses in the emulation to DML node names in
the virtual network description (these are not IP addresses). The emulator assigns IP addresses and
configures routing tables appropriately, consistent with CIDR hierarchical
address management. The mgrid system uses the FWTB
translations to inject/extract application messages to/from the network
emulator.
Each line of the FWTB file
has the following format:
IPAddress/Netmask
Physical-Node Node-Type DML-Address
For example:
0.0.1.46/30 machine-23 Router 3:3(1)
This entry indicates that the DML
node with address 3:3(1) has been assigned the virtual IP
Address 0.0.1.46 with a subnet mask of 30. This DML node is hosted on the physical
node machine-23, and is functioning as a router.
There are two Node-Types, router and host. Applications and jobs run only on nodes of type host.
The mapping table contains more
detailed information for virtual hosts.
Each line of the mapping table has the following format:
Virtual-name
IPAddress Physical-Node CPU-percentage memory
For example:
n10.caltech.teragrid.org 0.0.2.40
compute-0-5 32 512
This entry indicates that the virtual host n10.caltech.teragrid.org is assigned IP address 0.0.2.40 and will run on physical machine compute-0-5. The virtual host will use at most 32% of total CPU cycles and 512MB memory.
There is another file that was generated: <GRIDNAME>-app.xml, which defines the mapping information in XML format and is used by the scheduler. It includes a sequence of entry elements called a hostmap.
<hostmap>
<entry>
<v_host>0.0.1.10</v_host>
<p_host>compute-1-0</p_host>
<speed>90</speed>
<memory>512</memory>
<gatekeeper>6767</gatekeeper>
</entry>
...
</hostmap>
This hostmap has an entry, which specifies that jobs for virtual host 0.0.1.10 will be executed on physical host compute-1-0 and will use 90% of the total CPU cycles and 512MB memory on that physical host. If the gatekeeper entry is present, a port number must be specified. The daemon startup script will automatically start a gatekeeper daemon on the indicated virtual machine and port number.
|
See: $MGRID_ROOT/examples/template_mgrid_globus/camps8-app.xml
for an example. |
5.3 Application
Configuration
To configure your application for mgrid_globus you will need to create an XML job description file and a Globus RSL file under your