HP Gen 8 Speed Problems Solved

Well, not solved exactly. But ‘explained’ at least! (See the previous post for a description of the network speed problem that was annoying me).

My initial data copy from Server 1 (using SATA legacy mode) to Server 2 (using AHCI mode) finally completed, I finally got around to implementing the normal sorts of data safety checks you do with ZFS, including scrubbing my volumes (procedures for doing so, incidentally, I have now written up as a new Solaris article).

At this point I noticed something. Here’s server 2’s scrub progress report (AHCI):

[email protected]:~/logs# zpool status safedata
  pool: safedata
 state: ONLINE
  scan: scrub in progress since Mon Apr 25 16:56:31 2016
    734G scanned out of 7.32T at 360M/s, 5h20m to go
    0 repaired, 9.79% done
config:

        NAME        STATE     READ WRITE CKSUM
        safedata    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0

errors: No known data errors

I’ve highlighted the important bit as far as this issue is concerned: my zpool is managing 360M/second -that’s 360 mega bytes per second, which isn’t bad for a 4-drive array, when an individual SATA disk ought to be capable of doing around 100MB/sec, especially when you take into account that a scrub is doing a lot of computational work that has nothing to do with disk I/O.

And here’s the equivalent report from server 1 (using legacy SATA mode):

[email protected]:~# zpool status safedata
  pool: safedata
 state: ONLINE
  scan: scrub in progress since Mon Apr 25 17:35:31 2016
    48.4G scanned out of 7.32T at 169M/s, 12h34m to go
    0 repaired, 0.65% done
config:

        NAME        STATE     READ WRITE CKSUM
        safedata    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c2d0    ONLINE       0     0     0
            c2d1    ONLINE       0     0     0
            c3d0    ONLINE       0     0     0
            c3d1    ONLINE       0     0     0

errors: No known data errors

I think the fact that this box is managing only half the I/O rate of the other is the lion’s share of why the copy across the network was going so slowly: if server 1 can only fetch stuff from disk at the speed of a somnolent snail, it’s not surprising it can’t send it over the network at a decent rate either.

Tomorrow, then, I shall be purchasing a second el-cheapo PCIe SATA expansion card and rebuilding server 1’s zpool. We shall see what speeds I get then… watch this space.

Update: I assume that there must be a world-wide shortage of PCIe SATA adaptors, since I just spent my lunch hour wandering around the IT shops of Chinatown, including MSY, and being told at least 15 times that they were out of stock. Have ordered one from Amazon, which means I have to wait until mid-May before being able to sort this out. Frustrating!

The Gen, er, 8 shun Game

hpgen8As I mentioned recently, I have taken to installing Solaris 11 on my twin HP Generation 8 Proliant Microservers -but that doing so is trickier than it might be, because you can’t boot off a fifth drive if you run the server in AHCI mode (and if you run it in RAID mode, Solaris can’t identify any drives at all in the machine).

So I built my first Solaris server using Legacy Mode for the drives -and this sounded “bad”.

Well, it came time to build my second Solaris server -and before I did so, I took a trip to my local computer store and purchased a $30 el-cheapo PCIe SATA card. With that installed and the server put into AHCI mode, it was trivial to persuade the server to boot from the new controller card, thus permitting Solaris and AHCI to co-exist.

Except that I then began to copy my data from the first server (still in legacy mode) over to the second… at which time I noticed this:

ahci02

Now a transfer speed of 37.8MB/sec is not good. It equates to just 302 mega bits per second over a 1 Gigabit Ethernet connection: two thirds of my bandwidth appears to have disappeared.

Of course, you don’t get 130MB/sec (i.e., 1024 megabits per second) on Gigabit Ethernet at any time, because there are overheads…but at my achieved transfer rates: well, that’s a lot of overhead!

I also wasn’t too keen on the CPU figures I was getting:

ahci01

That’s essentially one CPU flatlined for the duration, which seemed a tad excessive for a simple file copying operation (albeit that I understand that using ZFS with raidz is essentially the same sort of thing as doing software NTFS RAID5 on a Windows box, which makes demands on your CPU too).

Remember, too, that when I first copied the data onto the legacy-mode first Solaris server, I easily achieved speeds of around 59MB/sec, or roughly 500 megabits per second, which is a bit slow but obviously about a third faster than this new server was managing to get.

So, first I enabled the write-cache in BIOS for the second server, changing nothing else and tried again. It made a dramatic difference to one thing, but not the other:

ahci03

With write cache enabled, CPU consumption goes way down, to a much more evenly distributed ~30%. But:

ahci04

…the network performance is still pretty much as it was: attrocious.

Next, therefore, I switched SATA modes in BIOS to legacy mode, just as the first server was already using. Regrettably, this causes the existing raidz zpool to be destroyed, so I basically had to start my entire data transfer from scratch, but was it worth it?

ahci05

Well, I’m a bit conflicted about the answer. Yes, definitely in that I’m now into the 42MB/sec range instead of the 38MB/sec place I was in before… but no, that’s not a huge up-tick in performance. (CPU remained in the 20 – 30% area, so no change there: enabling write-cache makes a big difference, clearly, though it obviously introduces some concerns for data integrity on a server).

At this point, I am left to wonder whether the fact that this new server has 4 x 4TB Western Digital Red drives compared to the original server’s 4 x 3TB WD Reds is responsible for the network throughput difference: you can only transport what the disks are able to pump out, after all. Maybe the 3TB drives fetch data more slowly than the 4TB ones, so when the 4TBs were the source pumping to the 3TBs, I got better speeds than when, as now, the 3TBs are pumping to the 4TBs??

Nice theory, shame about the facts. The spec sheet for the entire family of WD Reds indicates that the 3TBs and the 4TBs are identical except for a 3MB/sec difference in “internal transfer rate”. I don’t see how that converts into a >16MB/sec difference in network speed I’m seeing here.

Frankly, I can’t work it out. Maybe it’s the network cables I’m using (but I’m not prepared to re-cable my patch panel to find out!) Or maybe it’s sunspots and wind direction. Who can say?

Does it persuade me to shun the Generation 8 servers as a Solaris platform? (And I apologise to my UK readers of a certain age for the terrible pun in this blog’s title. Be grateful I didn’t give you a twirl. Or a conveyor belt).

Not really: if you make a decision about write cache enabled or disabled to get your CPU levels down (keeping in mind the potential data-safety costs), a $30 SATA card will give you AHCI capabilities for your data drives; and whilst AHCI doesn’t seem to do a lot for me on the plus side, a slower data transfer speed seems a price that’s probably worth paying: you only load these babies with bulk data once, after all.

The lesson I really take away from it is really that pursuing specific numbers is a mug’s game.

For I am possessed of a cat…

This is Harper, one of our two rescue cats:

cutecat01

Cute, isn’t he? Very fond of laps, he’ll sit there all night if you let him.

Just don’t try and pick him up, though. I did, and I immediately came to regret it:

catclaw02

The other arm’s bleeding, too. I’m not entirely sure why it took me quite so many seconds to let go of him, but I wish I’d been quicker!

Furriners!

So, last time I promised a tale of woe and skulduggery regarding my Solaris Samba shares.

Let’s assume I’ve created a new share, called “bulkdata”, on a freshly-built Solaris box as per this article.

On Windows, I can connect to the share without problem and set about creating a new text file like so:

lorem

Note that the file is called Götterdämmerüng.txt, so it contains three pesky ‘foreign characters’ to start with. The contents of the file are also riddled with foreign characters: genügte as the seventh word, for example. Then it goes completely bonkers as I slip into fluent simplified Chinese!

As you save that in Windows 7, Notepad gives you some options:

lorem02You can choose to save in ANSI (the default), Unicode or variations on Unicode. Given what my Solaris box is telling me:

lorem03…it would seem that selecting that ‘UTF-8’ option would make sense, though it’s a bit of a pain that it’s not the default. Fat chance of getting ToH to remember to alter the encoding every time a new file needs copying to this server! But I’ll give it a go anyway… UTF-8 it is.

Once saved, here’s what I see on the server itself:

lorem04

Excellent news! The file is there, complete with its set of three foreign characters. We have end-to-end foreign character awareness, with not too much effort!!

Or do we? Let’s just check the contents of that file, shall we?

lorem05

Well, genügte is there, in all its umlaut-y glory, so we certainly have end-to-end European foreign character awareness… but sadly the Chinese will have to use some other Samba share, because this one clearly chews this stuff into oblivion.

Or does it!

Assume for a moment that I’ve just had a mental breakdown and decided that I prefer using vi to nano:

lorem06

Different text editor…same file…same contents… only now the Chinese is perfectly cromulant (and the German remains so too).

And just for laughs:

lorem07

So the operating system’s own cat command also has no trouble displaying the simplified Chinese, nor the accented German. Just nano, apparently.

How about I step back a bit, and save that file in the default Windows format, though? That would be “ANSI”. I’ll save the same file in that format in Notepad as götterdämmerüng2.txt. Doing so prompts Windows to warn you:

lorem08

…but if you persist by clicking [OK], the file arrives at the server in one piece and looks pretty good in vi:

lorem09

OK, we really have lost the Chinese at this point, but the German is still sufficiently umlaut-y for me to be happy. Except it’s vi, and I prefer nano:

lorem10

…and nano loses the umlauts and the Chinese.

And the OS itself is none too happy, either:

lorem11

It seems to know there’s something where there ought to be an umlaut, but can’t bring itself to display the right character at the right time.

The same thing happens in Linux, by the way, if it happens to be a client of the same Samba share: nano glitches on the umlauts (and loses all the Chinese) whereas vi copes with the umlauts (but also loses all the Chinese… ANSII is not good for that sort of character set!). Linux’s cat command also behaves the same way as nano: umlauts appear as strange question-mark lozenges.

This sort of things has bugged me for more years than I care to count (OK, for 18 years if we have to, since I first started ripping Italian and German opera to MP3 files that had to be named correctly, in 1998). I never ceased to be amazed at how the damned problem still hasn’t been sorted out completely as yet: UTF-8 was supposed to make this stuff go away a long, long time ago.

For many years, my battle was Linux v. Windows: Windows file names containing accented characters would be garbled on the Linux end of things, unless I remembered to add the ‘utf8’ mount option in /etc/fstab. It only took me 5 years to work that one out!

But Solaris is a bit more insidious even than that: it gets the display of things right at the OS level straight away… but then scrambles the contents, depending on what application you choose to use!

I guess the moral is: watch what clients you use and how they save their data… and clearly, vi is the better text editor 🙂

HP Gen 8 and Solaris 11

hpgen8I bought a couple of these HP Proliant Generation 8 microservers last Christmas: they’ve been running perfectly fine since then, using Windows 2012R2, but I recently decided that although ReFS was probably OK for my needs, ZFS was almost certainly better: I care about my data and would hate to have it lost or silently rot away.

So I decided to install Solaris 11 (X86) onto one of them… and soon wished I hadn’t bothered!

The trouble is the way the server allocates its storage. It ships with four SATA slots, into which I’ve fitted my four 3TB hard disks. These I want to use purely as my data drives, so I have added an extra small SSD drive to the chassis so that it can be the O/S boot drive and leave my 4x3TB array alone for data serving purposes. HP thoughtfully provides enough SATA ports on its motherboard for this (but rather uncharitably doesn’t supply a fifth SATA cable to help you make use of it!)

In the BIOS, the server has three SATA controller configurations: RAID, AHCI and “legacy SATA”.

And here’s the kicker: if you use the RAID configuration, then Solaris doesn’t see any disks in the server at all (presumably driver issues which I’m not smart enough to sort out as yet). That particular installation was a non-starter from the get-go, therefore.

If you use the AHCI configuration, it does see the disks and will actually install to the SSD no problem… but the server will only boot up from the 4 data drives. Since the O/S was installed somewhere else, the thing won’t actually boot at all, instead sitting there twirling its thumbs in the hope that a network boot device will be found instead. There’s no option in the BIOS for telling the server to alter its boot order, either, so the problem is unfixable.

At my third attempt, therefore, I finally worked out that the controller has to be put into ‘legacy’ mode: that allows Solaris to see all the disks and to boot from the SSD …provided you also remember to go back into the BIOS and change the controller boot order so that controller number 2 (the SSD one) is listed before controller number 1 (the data drives one).

What a pallaver!

But I got there in the end, and the thing now works nicely enough. I don’t know what unintended consequences might flow from switching to “Legacy SATA mode” (sounds bad, right??!), but I’m busy copying 6+TB of data from the other server now and achieving sustained rates (using just one of the Gigabit Ethernet ports) of around 56MB/second (i.e., around 450 megabits per second, which isn’t too bad).

After that, there’s just the small matter of getting Samba sharing working on Solaris to figure out. I’ve mostly sorted that out, too (and written up some idiots’ guides to doing it and other, related home-server preparation tasks). But there are some interesting quirks… which I’ll save for another post!