Leave well alone


ZFS not being a viable option on Fedora, I wanted to create a RAID5 array using mdadm, formatted with XFS.

There are lots of suggestions around the web about how best to do the XFS formatting to achieve optimal performance.

Essentially, the optimzation process revolves around setting stripe unit and stripe width settings. What the “correct” values are for those things can vary according to who you read (and, more pertinently, what you expect to store on your array -lots of little files, or a preponderance of very big files (like virtual machine images, movies and so on).

On the other hand, there’s a school of thought that says XFS is reasonably smart and if you just let it default to automatically-determined values, things will probably be ok.

So I thought I’d check it out.

First, I created my array and let mdadm complete the initial ‘recovery’, so no background I/O was taking place. Then I formatted it as follows:

mkfs.xfs -f -b size=4096 -d sunit=512,swidth=1024 -L bulkdata /dev/md0

And then I benchmarked the array using Gnome’s built-in disks utility:

So not too shabby: 145MB/sec read, 69 MB/sec write and an access time of 15.5msec.

Then I reformatted the array like so:

mkfs.xfs -L bulkdata /dev/md0

Re-measuring the performance in the same way as before, these were the results:

This time, reads of 165MB/s, writes of 74MB/s and access times of 15.5msec. So, significantly faster reads, moderately faster writes and access times about the same: I’d say letting XFS work it out for itself is probably as good a strategy as any -for my hardware, at any rates.

I repeated the tests multiple times, and I also introduced different formatting options with different values for the stripe unit and width and blocksize: I could certainly get worse results, but I was never able to improve on the ‘just let it work it out’ results.

Your mileage may vary, of course. But me: I’m leaving my XFS array well alone and letting it sort itself out!

DIY NAS

I have spent most of the beginning of this year re-building my two NAS servers …and then re-re-building them because I wasn’t happy with their performance or some other configuration peculiarity. Not a speedy job when each rebuild requires you to move off 6TB of data and then copy it back when you’re ready!

My biggest problem was getting the RAID stipe alignment right: I cunningly used command line-fu to start partitions at various cylinders to try and get it right, but performance figures suggested I’d got it horribly wrong instead. So then I re-did them using the simple GUI tools provided… and everything worked beautifully. On this occasion, then, GUI 1, CLI 0.

But the job is finally done, and I now have a minimalist CentOS 6.3 purring away on both boxes, one being a mirror of the other and with decent read performance, which is the main thing.

I figured it would be useful not to forget some of these painfully-learnt lessons, so I wrote up the installation and configuration process as a new article.

At some point this year, I will start replacing the drives used with new 3TB WD Red drives… At least I’m now confident that things should be plain sailing at that point.

New Year NAS

It was six months ago that I got a couple of HP microservers, slapped Windows 2008 R2 on them, software-raid5′d 3 2TB disks in each and thus acquired, in total, about 8TB of usable, protected storage. It now being the New Year, it was time I had a bit of a re-think about those servers.

First, I wanted to bump up the storage. The servers come with only 4 disk bays (each), and one of them was occupied by the Windows O/S disk -hence there were only 3 bays for the ‘safe storage’ drives. Happily, I find I own a couple of 60GB solid state hard disks that are sitting on the shelf doing nothing -and it’s trivially easy to plonk them is as the boot disk, freeing up all four drive bays for safe storage duty. Even more happily, I had a couple of spare 2TB drives sitting around, so those newly freed-up drive bays could be put to good use.

So, each server now has a 60GB O/S drive, plus 8TB of storage (though if you RAID5 4 2TB drives, you end up with around 6TB usable storage). Per server.

So then it came down to a choice of OS and raiding technology. There were three basic options:

  • Stick with Windows 2008 (or maybe upgrade to Windows 2012)
  • Switch to FreeNAS and start using ZFS
  • Switch to CentOS and use an mdadm software raid, plus a traditional file system (like ext3 or ext4)

In the end, I decided to dump the Windows options, simply because it works, is dull and isn’t very educational. I don’t have moral objections to Microsoft these days, and Win2008R2 has done sterling service for the past six months… but I just can’t get excited about NTFS and dynamic disks anymore!

Much more fun was to back everything up very carefully (only I wasn’t quite as careful as I should have been… see below!) and then wipe one of the servers with a FreeNAS install. If you haven’t met FreeNAS before, I can certainly recommend it. Your server ends up running a console-only BSD O/S which you access and manage via a nicely polished web interface from a remote PC of some kind. Setting up new volumes as Raid-Z (ZFS’s new take on the basic RAID5 principles) was a matter of mere minutes, even as a complete beginner; and setting up Samba shares to expose the newly-protected storage was trivial. It’s slick and I was happy… for about 3 days.

The samba shares are there because ToH still insists on using Windows and the home entertainment consists of a Windows 8 PC sitting under the telly running Windows Media Centre. It was as I was trying to play music via these shares that I encountered a couple of rare ‘stutters’, where the music would pause until “whatever it was” cleared and sorted itself out. That had never happened before. I did some research and discovered various tweaks you can apply to a vigin FreeNAS install to make samba shares perform better -and, as far as I can remember, they worked well enough, but I would occasionally still encounter a stutter or two.

I don’t think I’m alone in having found that the N40L is a great little server that lacks a bit of puff, though -and once I read that others have had “issues” with it and FreeNAS, I’m afraid my mind got a little bit closed on the matter. The N40L is a lovely little server, but it’s not a performance giant …and asking it to cope with ZFS raid, even with 8GB RAM fitted, was probably a bit on the ambitious side. I’ll confess, too, that whilst FreeNAS seems a great way to do things, it’s new (to me) and thus has a learning curve: this being the Christmas Hols, and me having more inclination to Deck the Halls than battle with ZFS, I fear the pull of the tried-and-true began to outweigh the excitement of doing something sexily unfamiliar.

Hence, I ended up wiping the Windows 2008 off my second server and installing ye olde CentOS 6.3. A quick refresher on mdadm (the software RAID application) and I had 6TB of protected storage up-and-running using nothing more exciting than ext3 pretty quickly. I then wandered about, lost and forlorn, for a couple of hours as I struggled to remember how to set up Samba shares manually. I was failing miserably until I remembered that although I had edited my SElinux configuration to be disabled, I hadn’t actually rebooted the server afterwards, so the ‘enforced’ setting was still effectively in place. That which had remained unshareable for hours, after one quick reboot, became trivially easy to remotely access once more. (PS: I hate SELinux!!)

So, I now have one server running sexy FreeNAS -perfectly happily, as far as I can tell, though I am suspicious of its performance and suspect it might go horribly wrong at any moment. And then I have the other server running traditional CentOS -also perfectly happily, it would seem, with me basking in the glow of the comforting and familiar and sure that if it ever goes horribly wrong, it will only be because of something I typed.

The eventual plan, of course, is to have both servers configured identically and replicating amongst themselves with a scheduled rsync. So, ultimately, one of them will have to “win” and the other will become a mere clone of it.

Having to decide a winner is therefore a bit of a tricky one. In the red corner, there’s the fact that we watched The Dark Knight Rises on New Year’s Day streamed from the FreeNAS box, in high-def, with not one glitch, unscheduled pause or hiccough (though I did fall asleep after 50 minutes, so it’s possible the rest of the boredom actually streamed really badly, just without me noticing). Over in the blue corner, however, is the fact that a 1.5GHz AMD Turion is probably not up to running ZFS effectively, even with 8GB RAM to play with… whereas boring ext3 and a bit of mdadm probably suits that quite well. Difficult call to make, I think, without me spending a lot of time doing benchmarking I don’t actually have time to do…

There is one real nasty in all of this -and it’s an oldie and a goodie! Foreign characters in file names have been an issue once more. Take this bit of gibberish from an ssh session connected directly to the FreeNAS server:

[[email protected]] /mnt/SafeData/Music/classical/Richard Wagner# ls
./                                Parsifal/
../                               Parsifal (Levine)/
Das Rheingold/                    Rienzi/
Der fliegende Holl??nder/         Siegfried/
Die Meistersinger von N??rnberg/  Siegfried-Idyll/
Die Walk??re/                     Tannh??user/
G??tterd??mmerung/                Tristan and Isolde/
Lohengrin (Keilberth)/            Tristan und Isolde (Bernstein)/
Orchestral Music from the Operas/ Wesendonk-Lieder/

Anyone wanting to listen to a bit of Walküre or Götterdämmerung is going to be a bit disappointed, I think. However, maybe not, judging by this screenshot of the very same directory of the very same server taken on my Fedora-running PC:

Even Windows clients have no problem with that directory listing:

Similarly, when I copy exactly the same files from exactly the same external USB (formatted with NTFS by the original Windows 2008 server) onto the CentOS server, I have absolutely no problem with foreign characters at all… so somewhere during the FreeNAS construction process, I suspect there was a file system mounted without UTF8 being specified as one of the mount options. Since I know -from long and bitter experience- to do this on my Linux boxes without even thinking about it, I imagine this is one unfortunate result of my inexperience with all things FreeNAS and BSD! It is, therefore, another black mark against FreeNAS, despite it almost certainly being a result of a PEBKAC.

Anyway, I haven’t quite decided yet, but I suspect that CentOS will win out in this race, simply because I have other things to worry about at the moment.

Such as what operating system to run on my main PC this year? Yup: that’s still up for grabs, despite all the experimental, exploratory installing I did a few months ago. I spent some weeks with Windows 8, which wasn’t as horrible as I expected, but ultimately hit the boredom buffers, as often happens with me and Windows. Currently, as one of the above screenshot shows, I am running Fedora 17 with the XFCE windows manager (Gnome 3 being utterly beyond the pale as far as I am concerned). It’s OK, but I am encountering a hell of a lot of bugs, crashes and other assorted weirdnesses). So I don’t think it’s for me long-term.

Funnily enough, I reckon I know what I’ll be installing on my new 32GB RAM PC (when it finally arrives): CentOS 6.3. It is stable, cuddly (in a Gnome 2ish sort-of way) and functional, yet still requires copious quantities of command line affection from time to time. Sounds like all I could really ask for in a desktop OS! Roll on PC delivery!!

Until then, or the next time, Happy New Year to all my readers.