i-am.ws |
Network Speed with Zones
A little time back I was preparing for a big benchmark project where our customer wanted to compare a single large system using many zones with a more horizontally scaled infrastructure, consisting of a number of smaller servers, like V490 and V890. I immediately thought that replacing a number of servers, being chatty over the network, with a single server, carved up into zones, would give a big benefit in network performance. Zone-to-zone network traffic should be faster than server-to-server. So I fired off some emails to people that I thought would give me the final answer, but the responses were very mixed.
Therefore it was time to do some of my own experiments. Doing a big benchmark in one of the Sun Solution Centers, I had the availability of some serious hardware for these tests. On the other hand, as is usual with these types of projects, there was a lot going on at the same time, therefore in the end time was limited for this little exercise.
This was my test platform:
- A few 8 CPU / 16 core domains (1800 MHz US-IV+) on E25K.
- Couple of quad Gigabit Ethernet cards, connected to a SMC switch.
- We had to use the 'ce' network drivers, because there were incompatibilities with other ones.
This is the environment I built:
- domain A, zone 1, IP 10.1.1.131, interface ce2:1 - used for sending files
- domain A, zone 2, IP 10.1.1.132, interface ce2:2 - receiver, using the same physical interface, but with its own virtual interface
- domain A, zone 3, IP 10.1.1.133, interface ce3:1 - receiver, having its own network interface, but different from the one used by the sender
- domain B, zone 1, IP 10.1.1.101, interface ce4:1 - receiver, now a completely different domain, so will only communicate with sender over the copper wire
This provided us with three test scenario's: a) network traffic from one virtual interface to another, both on the same physical interface, b) two zones talking with each other, each with their own physical interface and c) two independent servers, or in this case domains.
I used ftp to send files of three different sizes: 1M, 3M and 1G bytes. All files were created in /tmp and sent to /tmp. I repeated each test three times. Here are the results (all times in secs):
| zone-to-zone same interface |
zone-to-zone other interface |
server-to-server other interface |
|
| 1 MB | 0.0072 0.0056 0.0058 |
0.0083 0.0077 0.0080 |
0.012 0.012 0.013 |
| 3 MB | 0.27 0.16 0.15 |
0.23 0.21 0.21 |
0.35 0.33 0.34 |
| 1 GB | 6.7 5.0 4.9 |
6.1 5.4 5.4 |
11.0 11.0 12.0 |
So, from this we can see clearly that zone-to-zone traffic doesn't "hit the copper" and probably gets shortcutted somewhere in the IP layer of the TCP/IP stack. I would think that with slower interfaces, like 100 mbps, the speed advantage will be even higher than the 1.5-2x we see here.
Posted at 01:11AM May 11, 2007 by WWWillem in Networks |
