Fast Messages Performance

FM on Myrinet

Bandwidth

Latency



FM on Giganet

Bandwidth

Latency



FM on shared memory (over a SMP's CPU/memory bus)

Bandwidth

Latency



Fast Messages Performance (Tables)

FM on Myrinet

Latency and Bandwidth

Message Size (bytes) Bandwidth (MB/s) Latency (usec)
4 0.95 9.63
8 1.91 9.49
16 3.83 9.59
32 7.55 9.72
64 14.15 10.02
128 25.97 11.42
256 43.09 14.85
512 63.89 20.32
1024 84.37 32.11
2048 100.27 56.37
2176 55.44 60.92
2304 57.73 63.64
2432 61.1 65.34
2560 64.05 66.74
2688 67.13 67.98
2816 70.6 69.58
2944 73.39 70.94
3072 76.7 72.49
3200 80.15 73.73
3328 82.99 75.45
3456 86.15 77.15
3584 89.64 78.92
3712 92.39 80.25
3840 95.73 81.8
3968 98.37 84.61
4096 100.1 87.74
4224 70.68 93.13
4352 72.75 93.99
4480 74.89 94.92
4608 76.94 95.81
4736 79.06 96.4
4864 81.31 97.08
4992 83.3 97.42
5120 85.22 97.65
5248 87.5 97.39
5376 89.67 97.68
5504 91.75 97.53
5632 93.84 97.67
5760 95.98 97.94
5888 97.83 98.43
6016 99.28 99.76
6144 100.49 99.38
6272 78.28 134.1
6400 79.7 134.25
6528 81.42 135.74
6656 82.89 134.83
6784 84.56 134.57
6912 85.77 130.38
7040 87.73 127.69
7168 89.23 125.31
7296 90.67 124.96
7424 92.07 125.31
7552 93.86 125.74
7680 95.25 126.16
7808 96.97 126.38
7936 98.14 126.56
8064 99.46 126.36
8192 100.21 127.51
8320 82.96 167.5
8448 84.12 169.59
8576 86.18 171.52
8704 87.37 172.02
8832 88.69 171.55
8960 89.89 168.24
9088 91.24 169.43
9216 92.54 168.33
9344 93.87 168.93
9472 95.01 171.17
9600 96.14 173.97
9728 97.45 175.28
9856 98.78 175.82
9984 99.99 175.76
10112 100.86 176.13
10240 101.54 177.22
10368 85.85 212.94
10496 86.74 212.26
10624 87.6 212.57
10752 88.96 212.6
10880 90.8 211.82
11008 91.35 195.19
11136 92.79 188.93
11264 94.08 187.38
11392 95.19 188.71
11520 96.15 188.22
11648 97.23 189.51
11776 98.37 189.84
11904 99.26 184
12032 100.38 186.77
12160 101.06 188.31
12288 101.66 188.1
12416 88.79 237.18
12544 89.61 238.47
12672 90.59 235.72
12800 91.55 237.34
12928 92.46 236.42
13056 93.26 223.57
13184 94.33 230.46
13312 95.29 230.32
13440 96.11 219.5
13568 97.04 222.16
13696 97.87 223.88
13824 98.76 223.16
13952 99.57 227.12
14080 100.71 228.25
14208 101.35 227.12
14336 101.86 223.89
14464 90.39 262.21
14592 91.29 259.12
14720 92.08 255.33
14848 92.72 249.87
14976 93.75 241.14
15104 94.52 240.74
15232 95.1 241.35
15360 96.03 243.32
15488 95.93 245.29
15616 96.73 246.75
15744 97.33 248.68
15872 98.22 249.79
16000 98.9 251.28
16128 99.67 254.17
16256 100.23 261.71
16384 100.53 258.82

FM on Giganet

Latency and Bandwidth

Message Size (bytes) Bandwidth (MB/s) Latency (usec)
4 0.95 14.7
8 1.7 16.07
16 2.82 16.5
32 5.91 16.04
64 8.15 16.01
128 15.11 16.87
256 28.04 19.01
512 49.73 22.95
1024 68.49 29.94
2048 80.95 44.29
2176 69.77 44.77
2304 70.9 44.84
2432 72.68 45.58
2560 73.05 45.41
2688 73.8 45.33
2816 75.36 45.2
2944 75.47 45.2
3072 76.2 46.41
3200 76.74 46.45
3328 77.3 47.69
3456 77.19 49.95
3584 78.4 49.72
3712 78.18 52.02
3840 78.88 52.81
3968 79.48 53.81
4096 79.87 55.44
4224 73.76 58.7
4352 74.28 59.6
4480 75.4 61.03
4608 75.5 61.79
4736 75.82 63.06
4864 76.62 63.23
4992 76.62 64.97
5120 78.13 65.98
5248 78.44 66.81
5376 79.02 68.48
5504 79.16 68.29
5632 80.04 70.14
5760 79.92 71.08
5888 80.4 72.82
6016 80.89 72.61
6144 81.13 72.78
6272 76.87 73.54
6400 77.16 74.18
6528 77.92 75.46
6656 77.84 76.08
6784 78.17 76.94
6912 78.64 79.04
7040 78.72 79.48
7168 78.8 81.14
7296 79.35 82.07
7424 79.59 84.03
7552 79.66 85.55
7680 80.25 86.15
7808 80.26 87.92
7936 80.52 88.61
8064 80.88 89.61
8192 81.32 90.85
8320 77.99 94.67
8448 78.13 95.36
8576 78.82 96.71
8704 78.63 98.47
8832 78.87 100.06
8960 79.32 101.28
9088 79.33 101.71
9216 79.47 103.42
9344 79.67 104.31
9472 79.7 106.08
9600 80.24 107.97
9728 80.68 108.29
9856 80.68 109.56
9984 80.59 110.86
10112 81.2 112.72
10240 81.25 113.33
10368 78.53 117.28
10496 78.63 117.72
10624 79.16 118.35
10752 79.27 121.23
10880 79.41 121.8
11008 79.74 122.54
11136 79.82 123.77
11264 79.82 124.77
11392 80.15 126.51
11520 80.11 126.97
11648 80.34 128.18
11776 80.69 129.54
11904 80.59 130.48
12032 80.78 132.35
12160 80.95 133.15
12288 81.48 133.91
12416 78.99 137.79
12544 79.32 139.41
12672 79.63 139.97
12800 79.69 141.35
12928 79.72 142.35
13056 79.91 143.71
13184 79.89 144.72
13312 80.07 146.5
13440 80.18 147.11
13568 80.44 148.76
13696 80.61 150.4
13824 80.86 150.76
13952 80.96 152.76
14080 80.99 154.06
14208 81.17 155.26
14336 81.35 155.88
14464 79.34 160.1
14592 79.48 161.44
14720 79.81 162.03
14848 79.81 164.15
14976 79.92 164.72
15104 80.31 166.18
15232 80.34 167.39
15360 80.31 168.04
15488 80.47 170.56
15616 80.57 170.77
15744 80.59 172.13
15872 80.87 173.25
16000 80.87 174.26
16128 80.89 176.32
16256 81.34 176.66
16384 81.46 177.91

FM on shared memory (over a SMP's CPU/memory bus)

Latency and Bandwidth

Message Size (bytes) Bandwidth (MB/s) Latency (usec)
4 1.72 3.31
8 3.45 3.32
16 6.88 3.7
32 13.38 3.7
64 26.41 3.33
128 53.97 3.45
256 86.27 4
512 122.75 5.25
1024 172.26 7.05
2048 222.86 10.13
2176 196.05 10.2
2304 193.83 10.11
2432 188.79 10.11
2560 196.87 10.09
2688 204.31 10.08
2816 205.44 10.31
2944 210.83 10.38
3072 215.8 10.43
3200 204.9 10.27
3328 214.19 10.23
3456 207.29 10.81
3584 227.2 9.99
3712 215.69 10.79
3840 224.56 10.57
3968 226.11 10.28
4096 234.33 10.24
4224 205.32 11.71
4352 200.49 11.75
4480 215.52 12.02
4608 213.75 11.83
4736 214.32 12.75
4864 213.17 13.15
4992 210.49 14.13
5120 209.43 13.38
5248 212.33 13.67
5376 220.53 14.88
5504 215.29 14.85
5632 220.93 14.96
5760 222.12 15.93
5888 219.6 15.33
6016 217.84 15.53
6144 230.78 15.63
6272 203.06 17.93
6400 204.63 17.25
6528 207.43 17.68
6656 205.53 18.34
6784 206.09 18.98
6912 209.48 18.63
7040 207.08 19.24
7168 210.95 19.56
7296 212.12 20.75
7424 213.64 20.87
7552 211.61 21.16
7680 213.66 20.71
7808 221.6 20.97
7936 209.19 21.44
8064 211.54 21.68
8192 214.38 21.58
8320 201.2 23.99
8448 201.19 24.18
8576 193.21 23.88
8704 188.57 24.63
8832 196.72 24.34
8960 198.28 25.4
9088 188.85 25.52
9216 186.23 25.89
9344 185.13 26.72
9472 198.65 26.51
9600 184.43 26.64
9728 196.13 27.53
9856 184.31 27.42
9984 186.81 28.38
10112 196.43 28.98
10240 188 28.68
10368 191.17 30.02
10496 179.87 30.85
10624 179.55 30.81
10752 183.12 31.23
10880 187.85 31.49
11008 188.48 32.45
11136 182.49 32.58
11264 190.14 32.68
11392 189.85 33.38
11520 190.79 34.46
11648 183.98 35.17
11776 183.59 34.04
11904 182 35.31
12032 193.11 35.17
12160 183.28 35.38
12288 182.4 35.85
12416 176.02 35.52
12544 177.95 36.65
12672 180.36 37.26
12800 180.79 36.7
12928 181.32 37.21
13056 188.69 38.23
13184 187.73 39.37
13312 190.35 39.01
13440 181.56 39.06
13568 191.37 37.55
13696 181.46 37.88
13824 191.29 38.82
13952 191.16 39.27
14080 184.13 38.41
14208 192.74 39.37
14336 184.22 41.06
14464 182.02 40.75
14592 179.48 42.77
14720 182.21 41.79
14848 190.67 41.5
14976 180.58 42.09
15104 183.42 41.83
15232 182.63 41.88
15360 183.42 44.22
15488 193.01 45.25
15616 192.31 44.65
15744 186.57 45.28
15872 184.54 46.15
16000 183.22 44.12
16128 193.53 45.35
16256 184.31 45.68
16384 195.65 45.9