# Infiniband Performance

In June, we remotely tested the performance of an Infiniband-connected cluster
owned by TopSpin. The cluster had 16
dual 2.4 GHz Xeon nodes, based on Tyan 2722S2-533 motherboards built with E7501 chipsets
(533 MHz FSB). Topspin two port 4x HCA's were used, connected to a TopSpin 360
Infiniband switch. The graphs below show comparisons with our current lattice
QCD cluster, which was built using 2.4 GHz dual Xeons on SuperMicro P4DPE-Q
motherboards with E7500 chipsets (400 MHz FSB), using Myrinet 2000 as an
interconnect (M3F-PCI64B-2 interfaces, M3-E128 switch).
All of the graphs shown below are also links. Click on any graph to download
an encapsulated PostScript version.

## Pallas Sendrecv

The Pallas Sendrecv
benchmark measures the aggregate bidirectional bandwidth
obtained using MPI_Sendrecv calls. We ran this test between two nodes on each
cluster.
We note that the measured Infiniband bandwidth is close to the 800 MByte/sec
limit commonly observed on PCI-X buses for bidirectional traffic. Also, the
new Myrinet M3F-PCIXD-2 interfaces are reported to sustain
489 MByte/sec summed bidirectional bandwidth, limited by the 2.50+2.50 Myrinet
hardware link layer.

## Netpipe

We used the MPI version of the Netpipe benchmark from Ames Lab
to measure the one-way bandwidth and latency of ping-ponged MPI send calls.
We ran this test between two nodes on each cluster.
On the bandwidth plot, we note again that the Infiniband result is close to
the PCI-X limit, and that newer Myrinet interfaces achieve very close to 250 MByte/sec.

Shown below is a Netpipe *network signature graph*, in which bandwidth
is plotted against transfer time for many different message sizes. The
intercept on the abscissa gives the zero-length message latency, and the
horizontal asymptote gives the saturation bandwidth.

We note from this graph that the latency for MPI messaging on Infiniband using
MVAPICH is 7 microseconds, and the corresponding Myrinet latency is 11
microseconds. Newer Myrinet interfaces are reported to have improved latencies.

## MILC Scaling

Shown below are scaling curves for the MILC improved
staggered code. On the two clusters, we ran a constant lattice size per node
calculation at various lattice sizes and on varying numbers of nodes. Each
curve shows performance for a given lattice size L^4, where L was 4, 6, 8, 10,
12, or 14. Each lattice size was run on these combinations of nodes: single,
2, 4, 8, and 16 nodes. Use a **wide** browser window to see these graphs
side by side. Again, click on the graphs to download encapsulated
postscript versions.
The increased performance on the Infiniband cluster is due to both the
increased memory bandwidth available with the E7501 chipset and the 533 MHz
front side bus processors, and the higher bandwidth and lower latency of
Infiniband compared with Myrinet.

Don Holmgren
Last Modified: 8th Sep 2003