I have spent most of the beginning of this year re-building my two NAS servers …and then re-re-building them because I wasn’t happy with their performance or some other configuration peculiarity. Not a speedy job when each rebuild requires you to move off 6TB of data and then copy it back when you’re ready!

My biggest problem was getting the RAID stipe alignment right: I cunningly used command line-fu to start partitions at various cylinders to try and get it right, but performance figures suggested I’d got it horribly wrong instead. So then I re-did them using the simple GUI tools provided… and everything worked beautifully. On this occasion, then, GUI 1, CLI 0.

But the job is finally done, and I now have a minimalist CentOS 6.3 purring away on both boxes, one being a mirror of the other and with decent read performance, which is the main thing.

I figured it would be useful not to forget some of these painfully-learnt lessons, so I wrote up the installation and configuration process as a new article.

At some point this year, I will start replacing the drives used with new 3TB WD Red drives… At least I’m now confident that things should be plain sailing at that point.

New Year NAS

It was six months ago that I got a couple of HP microservers, slapped Windows 2008 R2 on them, software-raid5′d 3 2TB disks in each and thus acquired, in total, about 8TB of usable, protected storage. It now being the New Year, it was time I had a bit of a re-think about those servers.

First, I wanted to bump up the storage. The servers come with only 4 disk bays (each), and one of them was occupied by the Windows O/S disk -hence there were only 3 bays for the ‘safe storage’ drives. Happily, I find I own a couple of 60GB solid state hard disks that are sitting on the shelf doing nothing -and it’s trivially easy to plonk them is as the boot disk, freeing up all four drive bays for safe storage duty. Even more happily, I had a couple of spare 2TB drives sitting around, so those newly freed-up drive bays could be put to good use.

So, each server now has a 60GB O/S drive, plus 8TB of storage (though if you RAID5 4 2TB drives, you end up with around 6TB usable storage). Per server.

So then it came down to a choice of OS and raiding technology. There were three basic options:

  • Stick with Windows 2008 (or maybe upgrade to Windows 2012)
  • Switch to FreeNAS and start using ZFS
  • Switch to CentOS and use an mdadm software raid, plus a traditional file system (like ext3 or ext4)

In the end, I decided to dump the Windows options, simply because it works, is dull and isn’t very educational. I don’t have moral objections to Microsoft these days, and Win2008R2 has done sterling service for the past six months… but I just can’t get excited about NTFS and dynamic disks anymore!

Much more fun was to back everything up very carefully (only I wasn’t quite as careful as I should have been… see below!) and then wipe one of the servers with a FreeNAS install. If you haven’t met FreeNAS before, I can certainly recommend it. Your server ends up running a console-only BSD O/S which you access and manage via a nicely polished web interface from a remote PC of some kind. Setting up new volumes as Raid-Z (ZFS’s new take on the basic RAID5 principles) was a matter of mere minutes, even as a complete beginner; and setting up Samba shares to expose the newly-protected storage was trivial. It’s slick and I was happy… for about 3 days.

The samba shares are there because ToH still insists on using Windows and the home entertainment consists of a Windows 8 PC sitting under the telly running Windows Media Centre. It was as I was trying to play music via these shares that I encountered a couple of rare ‘stutters’, where the music would pause until “whatever it was” cleared and sorted itself out. That had never happened before. I did some research and discovered various tweaks you can apply to a vigin FreeNAS install to make samba shares perform better -and, as far as I can remember, they worked well enough, but I would occasionally still encounter a stutter or two.

I don’t think I’m alone in having found that the N40L is a great little server that lacks a bit of puff, though -and once I read that others have had “issues” with it and FreeNAS, I’m afraid my mind got a little bit closed on the matter. The N40L is a lovely little server, but it’s not a performance giant …and asking it to cope with ZFS raid, even with 8GB RAM fitted, was probably a bit on the ambitious side. I’ll confess, too, that whilst FreeNAS seems a great way to do things, it’s new (to me) and thus has a learning curve: this being the Christmas Hols, and me having more inclination to Deck the Halls than battle with ZFS, I fear the pull of the tried-and-true began to outweigh the excitement of doing something sexily unfamiliar.

Hence, I ended up wiping the Windows 2008 off my second server and installing ye olde CentOS 6.3. A quick refresher on mdadm (the software RAID application) and I had 6TB of protected storage up-and-running using nothing more exciting than ext3 pretty quickly. I then wandered about, lost and forlorn, for a couple of hours as I struggled to remember how to set up Samba shares manually. I was failing miserably until I remembered that although I had edited my SElinux configuration to be disabled, I hadn’t actually rebooted the server afterwards, so the ‘enforced’ setting was still effectively in place. That which had remained unshareable for hours, after one quick reboot, became trivially easy to remotely access once more. (PS: I hate SELinux!!)

So, I now have one server running sexy FreeNAS -perfectly happily, as far as I can tell, though I am suspicious of its performance and suspect it might go horribly wrong at any moment. And then I have the other server running traditional CentOS -also perfectly happily, it would seem, with me basking in the glow of the comforting and familiar and sure that if it ever goes horribly wrong, it will only be because of something I typed.

The eventual plan, of course, is to have both servers configured identically and replicating amongst themselves with a scheduled rsync. So, ultimately, one of them will have to “win” and the other will become a mere clone of it.

Having to decide a winner is therefore a bit of a tricky one. In the red corner, there’s the fact that we watched The Dark Knight Rises on New Year’s Day streamed from the FreeNAS box, in high-def, with not one glitch, unscheduled pause or hiccough (though I did fall asleep after 50 minutes, so it’s possible the rest of the boredom actually streamed really badly, just without me noticing). Over in the blue corner, however, is the fact that a 1.5GHz AMD Turion is probably not up to running ZFS effectively, even with 8GB RAM to play with… whereas boring ext3 and a bit of mdadm probably suits that quite well. Difficult call to make, I think, without me spending a lot of time doing benchmarking I don’t actually have time to do…

There is one real nasty in all of this -and it’s an oldie and a goodie! Foreign characters in file names have been an issue once more. Take this bit of gibberish from an ssh session connected directly to the FreeNAS server:

[[email protected]] /mnt/SafeData/Music/classical/Richard Wagner# ls
./                                Parsifal/
../                               Parsifal (Levine)/
Das Rheingold/                    Rienzi/
Der fliegende Holl??nder/         Siegfried/
Die Meistersinger von N??rnberg/  Siegfried-Idyll/
Die Walk??re/                     Tannh??user/
G??tterd??mmerung/                Tristan and Isolde/
Lohengrin (Keilberth)/            Tristan und Isolde (Bernstein)/
Orchestral Music from the Operas/ Wesendonk-Lieder/

Anyone wanting to listen to a bit of Walküre or Götterdämmerung is going to be a bit disappointed, I think. However, maybe not, judging by this screenshot of the very same directory of the very same server taken on my Fedora-running PC:

Even Windows clients have no problem with that directory listing:

Similarly, when I copy exactly the same files from exactly the same external USB (formatted with NTFS by the original Windows 2008 server) onto the CentOS server, I have absolutely no problem with foreign characters at all… so somewhere during the FreeNAS construction process, I suspect there was a file system mounted without UTF8 being specified as one of the mount options. Since I know -from long and bitter experience- to do this on my Linux boxes without even thinking about it, I imagine this is one unfortunate result of my inexperience with all things FreeNAS and BSD! It is, therefore, another black mark against FreeNAS, despite it almost certainly being a result of a PEBKAC.

Anyway, I haven’t quite decided yet, but I suspect that CentOS will win out in this race, simply because I have other things to worry about at the moment.

Such as what operating system to run on my main PC this year? Yup: that’s still up for grabs, despite all the experimental, exploratory installing I did a few months ago. I spent some weeks with Windows 8, which wasn’t as horrible as I expected, but ultimately hit the boredom buffers, as often happens with me and Windows. Currently, as one of the above screenshot shows, I am running Fedora 17 with the XFCE windows manager (Gnome 3 being utterly beyond the pale as far as I am concerned). It’s OK, but I am encountering a hell of a lot of bugs, crashes and other assorted weirdnesses). So I don’t think it’s for me long-term.

Funnily enough, I reckon I know what I’ll be installing on my new 32GB RAM PC (when it finally arrives): CentOS 6.3. It is stable, cuddly (in a Gnome 2ish sort-of way) and functional, yet still requires copious quantities of command line affection from time to time. Sounds like all I could really ask for in a desktop OS! Roll on PC delivery!!

Until then, or the next time, Happy New Year to all my readers.

Storage Solutions

Long-time readers may recall that I bought at least one of these:

It’s a Drobo and it’s been a disaster. First, because it’s temperamental: move it, or power it off then on, or basically breathe anywhere in its general direction, and it will dramatically start flashing red lights, indicating total failure (and lost data).

When you have suppressed the sick feeling in your stomach at that thought, you can power it off and on a few more times and, probably, it will decide to reboot in ‘green light’ mode. At which point your data is safe for as long as you don’t breathe again.

Secondly, it’s incredibly slow to re-protect your data. One of the advertised joys of the Drobo was that you could eject one of the four existing hard disks, pop in a new one of greater size and then just sit back and wait for the thing to re-distribute your data over the whole array so that it was all protected once more. Which is, indeed, what happens… except that you wait a long, long, very long time. And throughout the duration of that wait, your data isn’t protected against another hardware failure at all. When I last tried to do this with 4 2TB drives, the box spent 6.5 days thinking about it.You run with unprotected data for a week and see how you get on sleeping!

And third: the Drobo was and remains relatively noisy. Easily audible above the noise of a loud action movie it’s providing the data for, that’s for sure.

We had a council clean-up this week, so my Drobos finally met their destiny as land-fill, as should have been the case many years ago. I hated them, and I’d never buy another. Good riddance to bad rubbish, I say!

Let me instead introduce you to this beauty:

It’s an HP Proliant N40L Micro Tower Server. It shipped with a single 250GB hard disk, but with space for three others of any capacity. It also shipped with 2GB of ECC RAM and an AMD Turion II running at 1.5GHz.

For that lot, I paid the princely sum of AU$264, from these guys, who were the cheapest I could find. I notice that today, they’re quoting an extra $25 for the same config, so obviously I got lucky!

I added 8GB of ECC RAM -and fitting it was not fun. It requires pulling the entire motherboard out, and that can only be done by unplugging all power, SATA and other cables. Those cables fit tightly in the confined space available -and the SATA connector, in particular, seems to have been welded into place by cruel people with exceptionally fine welding skills. My previous experience of detaching soldered SATA connectors from motherboards came back to haunt me… but we got there in the end. I’ve also added a USB-3.0 expansion card to fill the available PCIx slot, but I didn’t bother adding a graphics card, so it’s strictly on-board graphics for me.

All up, including postage, I think one of these things cost me AU$370. So naturally, I got two. And that still means I’ve bought two brand new, quality servers for hardly much more than I paid for one of my original Drobos!

Of course, you only get 250GB of storage for that price: you have to fill the other three slots (non-hot-swappable) yourself. Fortunately, I had quite a few 2TB drives knocking about the place, thanks in part to me having recently destroyed Drobos en masse. So, pop those in (very easily!), and I now have 6.2TB in each server.

Time then to install a server OS… and it could have been Scientific Linux, of course. But since I have my Technet subscription, and I wouldn’t mind learning more about Windows 2008 R2 administration, on that goes instead. Turn the three 2TB drives into a single 4TB, RAID-5 array… and I now have 8TB of protected storage, humming very quietly in the background. I think each server consumes about 50W, which seems economical enough. Plus, the big bonus: I can barely hear them, even though they sit just behind my triple-monitor setup, precisely at ear-height.

A 1.5GHz CPU doesn’t sound like much: I’ve seen people slagging the Turion IIs off as though they were barely in the Intel Atom class. But Windows 2008 is certainly responsive when I remote desktop to it, and both boxes are running virtual machines (one each) without seeming to struggle. I’ve no complaints about them, anyway: fast enough for this old-timer.

Essentially, all I’ve ever wanted is for near-silent, RAID-protected storage for my music and other multimedia collections. The Drobos failed to provide that on so many levels, it wasn’t funny. These new HP boxes, in contrast, do the job just fine… and give me a capable, stable platform to run permanently two large-ish (6GB RAM and 200GB HDD) virtual machines at the same time, which is a nice bonus. At AU$264, I’d recommend them to anyone, though at AU$289, I find even my enthusiasm waning a bit.

But I still like them a lot, and they have cheered me up no end, allowing me to dump the damn Drobos. That’s worth almost any price, come to think of it.

Quickie NAS

I never actually used Windows Home Server (WHS), but I thought about doing so often enough. It’s killer feature (for me)? The ability to plug in different disks, of different sizes, from different vendors (and even using different interfaces -I have a lot of old PATA drives kicking around!), and have them appear to the rest of the world as one large storage ‘pool’, with in-built redundancy. This was called ‘Driver Extender’… and has just been removed as a feature from the new Version 2 Home Server product. It seems a bit of a weird decision on Microsoft’s part, removing one of the two key product differentiators that made Home Server special. It wouldn’t surprise me to see the entire thing killed off, to be honest.

Anyway, the reason I no longer care too much about WHS is that I have my own way of doing networkable, extensible bulk storage: a Drobo with a WDTV Live media player.

My particular “first generation” Drobo only takes 4 hard disks -newer and more expensive ones can take up to 8. But using 2GB drives, that still means 6TB of usable storage (4 x 2TB, minus 2TB used for data protection), which is enough to be going on with. If 3TB and 4TB drives ever make an appearance, I’ll probably be a firmware update away from being able to increase my protected, usable storage space accordingly. Regardless, you can stick any combination of SATA drives into the Drobo you happen to have handy, and swap out smaller drives for bigger ones as your storage needs grow (and as your wallet finds it can cope). There’s no networking capability (you’ll need to pay stupid money for the Drobo Share, or the Drobo FS to get that), but you do get extensible, protected, set-and-forget storage that more-or-less just works (see below for the ‘more-or-less’ bit!).

The WDTV Live is a good media player. I had a plain old WDTV before the ‘Live’ version came out, and the upgrade gets you a networked media player. Set it up with an IP address, plug in an ethernet cable and it immediately makes itself visible as a Windows (well, Samba, anyway!) Share on the local network. Other than that capability and a slightly slicker front-end, there’s not a lot of difference: the thing is still capable of playing just about any media format you throw at it, has no problem with High Definition content, has a lovely “ten foot interface’ that anyone can drive within seconds… and just works, beautifully.

Stick these two products together, then, and what do you have? Basically, the Drobo just plugs into the WDTV Live via USB, is then seen as a single giant volume full of multimedia files… and the contents of that volume are then shared around the rest of the network, thanks to the Samba-sharing nature of the WDTV. When I rip a new CD on my PC in the study, therefore, it’s trivial to copy the output to the Drobo sitting under the TV on the other side of the house, despite the Drobo not having ‘intrinsic’ networking of its own. So what you actually end up with is a NAS that does excellent duty as a media server and player. Someone should design a product that includes both bits of functionality in the one box!

The networked Drobo FS costs about AU$850. The standalone Drobo Share costs AU$300. Neither would be able to play a bean on my TV! My non-networked Drobo cost AU$599, and the WDTV Live cost a further AU$189… so I end up with viewable, networked, extensible, protected storage for AU$788 instead of non-viewable, etc, etc for AU$850 – AU$900. (Hard disks cost extra, of course).

I’d thoroughly recommend the WDTV Live… it’s really plain sailing to use, and you couldn’t get a more capable, simpler media player. We ditched the original WDTV player a year or so ago for the joys of Windows Media Center running on a spare PC… but the usual Windows problems meant that experiment turned into something of a disaster (crashes, driver problems, forever updating etc etc). We were so pleased to be able to junk the complexities of Windows for the highly-functional simplicities of the WDTV once more!

I wish I could recommend Drobo quite so unreservedly. If you’d asked me three months ago, I would have done. But since then, I made the mistake of purchasing a new one for an elderly friend. I mention the ‘elderly’ bit because his requirements, above all, are for something that simply works, without fuss, bother or the need for constant fiddling and tweaking. He is a very non-technical person, and his movie collection needs to be safe without having to think about it. A pity, then, that the Drobo unit I purchased for him turned out to be defective: it didn’t work at first, it then worked long enough to copy a couple of terabytes onto, and then it decided not to work again once it had been plugged into a different PC. It would hang during its boot sequence; it would declare it couldn’t find some disks, then decide it could see those after all but now couldn’t see the ones it had no problems seeing before; it would not be detected by Windows 7 at all, and then it would be detected without a problem, until you rebooted the PC -after which it would revert to being undetectable. It was bonkers, frankly. Precisely what you don’t want when you buy ‘safe, protected, reliable storage’!

Naturally, you get the odd lemon turning up whenever you take the hardware-purchasing plunge, but I can tell you: getting one lemon makes you have second thoughts about the earlier purchase that has never put a foot wrong! It just undermines your confidence in the product as a whole, in short. And it doesn’t help that their “support desk” has the same senseless, robotic and dumbed-down attitude that all support desks seem to go for these days. All I wanted was a returns authorisation number. Instead, I get asked to produce a diagnostic log. Fair enough: I try to do that, and I can’t because the unit is completely unrepsonsive. I tell them this. They reply not with a ‘Jeez, it’s screwed then!’ but with a ‘well, can you try using the Firewire port instead of the USB one’! I don’t even have a Firewire port, I point out. Well, I’ll need to escalate this to the next level of support, I am told. No you won’t, I say… either authorise the return right now, or I start consulting lawyers. At which point, the return was authorised without further comment!

I don’t like the fact that I had to wrestle with them like that. The second I couldn’t produce a diagnostic log because the unit had hung, they should have authorised the return. The suggestion to plug it into a Firewire port reflects the fact, I think, that Drobos are very popular in Apple Mac circles… and I imagine the dumbed-down, treat-you-like-a-moron, have-you-tried-turning-it-back-off-and-on style of support is designed to cater to that particualr type of audience. It didn’t do anything to endear me to them or their product, though! (Can you tell??!)

Anyway, I’d like to say that the guy who actually sold me the thing couldn’t have been nicer or more solicitous: he’s gone out of his way (literally: he turned up at the office today to pick the defective unit up personally) to see me right with a new Drobo that works properly. Time will tell on that score, I guess, but I can’t really fault his efforts thus far. Meanwhile, my own, original Drobo sits there quietly under the telly doing sterling service without the slightest issue. So yes, on balance, I would still recommend it. Just make sure you get an excellent vendor -and don’t waste too much time with their useless technical “support”. My vendor, by the way, who comes highly recommended, is Ross at Ineedstorage.