Dizwell Informatics

News from Nowhere

Bonding Network Interfaces on Solaris

My HP Proliant Microservers ship with 2 Gigabit Ethernet ports: it therefore seems a bit silly to only be using one of them at a time! Why not “bond” the two into a single 2-Gigabit interface and double my network transport speeds for free?

In Solaris, this is called “link aggregation” (it’s probably called that in a lot of other OSes, too!) and it’s quite easy to set up.

First, make sure your interfaces aren’t already configured by running the command (as root):

dladm show-link

That will show you something like this:

root@rvw:~# dladm show-link
LINK                CLASS     MTU    STATE    OVER
net0                phys      1500   up       --
net1                phys      1500   up       --

Anything listed as “up” has already been configured, so my two interfaces have to be destroyed before an aggregation of them can be created:

ipadm delete-ip net0
ipadm delete-ip net1

And with that done, I can now create the new aggregate link:

dladm create-aggr -l net0 -l net1 bond0

(The name is ‘bond’ just because I felt like it. You can call it pretty much anything you like).

Next, you create an interface for the new link:

 ipadm create-ip bond0

And finally you just assign an appropriate network address to the new interface:

ipadm create-addr -T static -a 192.168.137.1/24 bond0/v4

Assuming you want the server to be able to see the Internet, too, you will additionally need to specify a default gateway and DNS server address for the new aggregated interface. This gets the default route correct:

route -p add default 192.168.137.69

And to specify a DNS server, you just do this:

svccfg -s network/dns/client setprop config/nameserver = net_address: "(192.168.137.69 208.67.222.222 208.67.220.220 8.8.8.8)"

(That’s all one one line: I am specifying 4 space-separated DNS server IP addresses: first is my own internal one, then come the two for OpenDNS and finally comes one for Google’s primary DNS server).

Confirm your changes have taken effect with the command:

svccfg -s network/dns/client listprop config

…and you should see something like this:

root@rvw:~# svccfg -s network/dns/client listprop config
config                      application        
config/value_authorization astring     solaris.smf.value.name-service.dns.client
config/domain              astring     dizwell.home
config/nameserver          net_address 192.168.137.69 208.67.222.222 208.67.220.220 8.8.8.8

That should do it: now your two independent interfaces should work as one.

I will confess, however, that it was only at this point that I discovered that as I hadn’t configured link aggregation on my second HP server, the transfer speeds between the two of them didn’t go up by much: from about 50MB/sec to around 60MB/sec. To see a real doubling of speed, I’d have to aggregate on both ends of the link!

Additionally, my hard disks aren’t the fastest things on the planet… so even with aggregated links everywhere, I suspect they’d turn out to be the real, real limiting factor.

And finally… there’s a bit of a sting in the tail. If all you are measuring is the speed of transferring data between two servers, the default load-balancing algorithm used in the sort of aggregated link the above commands will create won’t actually help you much at all! Load-balancing works by deriving a hash from the header of the packets being sent -and in a two-server setup, the source and destination will always be the same, so the hash will always be the same, too. In other words, as far as traffic between server A and B are concerned, it will always use either the net0 or net1 components of the bond0 aggregated link, never both.

There are different ways of aggregating, though: a round-robin approach would send one packet via one component interface and the next by another. That really would start to consume bandwidth on both interfaces at once, even for simple server A-to-server B jobs. But it does so with a cost: namely that it permits packets from one interface to beat the arrival of a packet sent on another interface that was meant to arrive first. Out-of-order arrivals mean the client has to spend CPU time re-ordering them and at that point, you’re slowing things down once again (and sending your CPU consumption levels higher than they used to be, too).

So what does link aggregation really get me? Not the doubling of throughput of simple server-to-server traffic, that’s for sure. But it does mean that if someone is watching a movie on the TV computer at the same time as I’m uploading a new batch of opera tracks on my desktop PC and the between-servers rsync job suddenly kicks in, my network should cope: now multiple clients are involved, the hashing algorithm used by load-balanced aggregation should mean all the various network traffic gets distributed across different physical interfaces, not saturating any one in particular. So it’s still worthwhile my doing it. Probably…