Oracle on Ubuntu 16.04

mandelaWith Ubuntu 16.04 as my new desktop, it was inevitable that I would need to cook up one of my pre-installer scripts to help automate the installation of Oracle 12c onto it.

Like the pre-installers for various other distros and O/Ses, I’ve given my script a name… and “Mandela” was the obvious choice. Ubuntu is a South African word that means ‘oneness of humanity’ (very roughly!). Mandela was intimately aware of it and was a great exponent of it. It also meant a lot to Mark Shuttleworth, the founder of Canonical, developer of Ubuntu, the O/S. It’s the reason why he called his distro ‘Ubuntu’ in the first place.

So anyway: I’ve put together an article on how you use mandela (the Oracle preinstaller) to make installation of 12c on Ubuntu relatively painless. The script is available for download either here or here.

Additionally, I found I wanted to install SQL Developer on Ubuntu, so there’s now an article for that, too.

Xenophilia

Picture the scene. My HP server has just had its set of 4 x 3TB disks replaced, so there is a pile of 3TB disk sitting there, spare, on my desk.

Around the same time, as a result of seeing a work colleague’s high def monitor, I decide that I want one too. The annual hardware budget having been blown getting the new 4TB disks for the server, however, the chances of me soon acquiring a shiny new monitor of whatever definition are exceedingly slim. But I have a degree in cunning, so I decide to install a new NVidia graphics card into my OpenSuse-running main PC to prepare it to drive a shiny new high-def monitor. At which point, OpenSuse chokes, explodes and otherwise kicks the bucket: where previously my OpenSuse desktop had been glitz and glamour personified, now the drop-shadows around windows have gone; wobbly windows have stopped wobbling; the desktop cube has started pining for the fjords; and, generally, the entire OS I’ve been happy with for 8+ weeks now looks completely awful.

Faced with a spare pile of hard disks and a screwed-up PC O/S install: What would you do?

Me? I decided to swap in the four drives to make a new RAID array for my PC and take the opportunity whilst I’m at it to ditch OpenSuse Leap 42 and experiment instead with Ubuntu 16.04. This was probably a bit perverse on my part, since I’ve been delighted with OpenSuse for most of the time I’ve had it, and generally regard it as pretty wonderful. So parting with it was a bit hard -but inevitable, given its new-found graphical screwiness. Besides, I’m a sucker for trying new things out and Ubuntu 16.04 (codename: Xenial Xerus, apparently meaning, roughly, “friendly squirrel”) was released to the world just three weeks ago and most of the reviews of it sounded interesting.

I last used Ubuntu back in 2006 and/or 2008, so it’s been a while! I swore off it when they introduced Unity, the ghastly launcher-thing that sits on the side of your screen, vertically; that always struck me as determinedly awkward, given most people have horizontally-oriented monitors. I only considered Ubuntu 16.04 because, for the first time ever, the Unity launcher can be configured to display horizontally, where it makes much more sense and looks just like the launcher of any Windows or “normal” Linux distro you care to mention.

The other real reason for thinking of Ubuntu today? In a word: ZFS. Ubuntu integrates ZFS tightly into itself in a way that OpenSuse never does. So where on Suse I had to script the importation of my music zpool after every PC reboot, Ubuntu just mounts it automatically without me (almost) having to lift a finger. ZFS is important to me for data integrity reasons: a Linux distro that treats it as practically a native file system gets my interest.

But I’m getting a little ahead of myself: first, I had to install the thing. And I have to say, it was an atrocious installation process. I was installing from a DVD I’d burnt from a downloaded 16.04 ISO. Every time I tried, it kept stalling on the ‘file 60 of 60’ mark. By (1) not selecting the option to update software as part of the install, nor to install third-party multimedia codecs; and (2) manually configuring a wireless network to my mobile phone acting as an access point, I finally managed to get this to stop being a problem. But even when, under these circumstances, the install sailed through to completion, the OS would fail to reboot properly when prompted and I was required to manually switch the PC off and on again. Short version: it was long, slow and flakey.

I subsequently tried installing it on a variety of old PCs, spare laptops and virtual machines …and the story was completely different. So it must just have been bad luck (or poor install media: those other installations were done from a bootable USB stick, not the DVD). But it left a poor impression, anyway.

So what’s it like when finally installed?

Well, for starters, I find Ubuntu to be nicely presented (once you change the slightly-hideous purple desktop background!), with a good choice of default fonts for the GUI and the terminal. OpenSuse is attractive in quite a flat way (the modern æsthetic, I guess), but Ubuntu feels attractive in quite a normal way by comparison. The only two exceptions to this are (1) the windows decorations (minimise, maximise, close) are on the left-hand side of the window, not the right and this cannot be changed in 16.04; and (2) Ubuntu’s use of ‘global menus’ and/or menus in the toolbar of application windows is pretty bad.

The global menu is just madness: if you’ve got a text document open in the bottom right-hand corner of your 32″ screen, you have to navigate all the way up to the top-left of the screen before you even see a ‘File’ menu. No thanks. Happily, in System Settings -> Appearance -> Behavior you can click the option to ‘show the menus for a window… in the window’s title bar’ and that problem is fixed. But don’t close the appearance configuration tool just yet, because another annoyance with Ubuntu’s application menus can be sorted whilst you’re here: by default, they remain hidden until you mouse over the application window’s header bar. Even when you hover, there’s a momentary wait before the menus appear… it’s jarring and slightly unnerving. It also means you have to aim your mouse at ‘nothing’, hoping you end up in the vicinity of the ‘File’ menu (or whatever you were hoping to hit). Chances are, you will miss, most often by quite a large amount! It seems a preposterous way of doing things to me. Fortunately, in the Appearance tool, there’s the option to set ‘Menus visibility … Always displayed’ -and then, finally, Ubuntu behaves as most other operating systems do!

So that is all fixable, but the windows controls are permanently placed where Mac users and southpaws will enjoy them …and I don’t! I am hopeful that the ability to move the window controls back over to the right will soon return. It was certainly possible to put them there in previous Ubuntu versions; hopefully it’s just a tweak or twiddle that will do the job once someone, somewhere figures it out…

As a bonus, Ubuntu loves my new graphics card and wobbles its windows alluringly and spins its desktop cube speedily, which makes me happy. Graphically, it’s pretty slick. Its choice of wallpapers and other theme-able elements is a bit lacking, however (Xubuntu 16’s selection is a lot nicer, in my view. Fortunately, it’s trivial to download the Xubuntu ones and back-load them onto Ubuntu by copying the contents of /usr/share/xfce4/backdrops to /usr/share/backgrounds, as root).

What Windows users would call the ‘start menu’ is, in Ubuntuland, called ‘the Dash’. I find it a bit of a problem, since it doesn’t present a nice hierarchical menu of application types into which you can drill at leisure. So you don’t get a small, simple, all-over view of what programs you’ve installed. Instead you get a ‘home page’ that lists recently-used applications and files/folders, which might be moderately useful if it displayed more, but its icons are huge and the data density is exceedingly low.

To see anything more, you have to click various icons at the bottom to activate various “lenses”: there is a specific one for searching for videos; another for music; another for files… and there’s one that looks like an artist’s easel which is actually an ‘A’ for applications. That’s what will show you all your installed apps in one go, provided you (a) go on to click the ‘Installed’ option rather than accept the default recently used/installed/dash plugins view; and (b) don’t mind huge icons and a lot of scrolling.

I don’t find it very useful; I rather think you’re supposed to type what you want in the search filter instead (but if you want to compose music, you have to know to type ‘m…u…s…e’ for ‘musescore’, which I don’t think works for a lot of things whose specific names I neither know nor remember).

There seems no easy way to alter the size of the Dash icons (changing the size of the launcher icons (i.e., the icons in what Windows would call the ‘quick launcher’) is a piece of cake: System Settings -> Appearance -> Look -> Launcher icon size. But that does nothing for Dash icons, leaving the Dash an extremely poor way of launching things (for me, at least: your mileage might well vary, of course).

By default, Ubuntu only enables one virtual desktop (a curious choice in my view: I regard virtual desktops as one of Linux’s best features). But it’s easy enough to switch them back on, at which point the Workspace Switcher button becomes meaningful. Sort of…

The equivalent 4-pane switch in OpenSuse lets you click on any of the quadrants to immediately switch to that virtual desktop; and if you’ve got a Desktop Cube effect switched on, the desktop will be ‘travelled to’ by spinning the cube. But not so in Ubuntu. Click one of those panels in the switcher button and you get this:

So I now have to click one of the displayed virtual desktops to switch to it. No Cube effect kicks in, and I’ve had to use my mouse once more than necessary. I wasn’t impressed.

Whether it’s a failing of Gnome or Unity or just of Ubuntu, I couldn’t say, but the notification system seems weak. In point of fact, there don’t seem to be many notifications in the first place, but when they do arrive (to tell you that there are software updates available, for example), you can’t do anything with them. You’d think (well, I would anyway!) that you would be able to click the notice to launch the software upgrade process, but you can’t. Instead, the notice just fades into a blur if you hover your mouse anywhere near it, so that you can’t meaningfully click it at all. You can’t dismiss the notice, either: it just sits there, pointlessly, until it decides to vanish in its own good time. I was fortunately able to install CustomizeNotifyOSD to fix that last issue (now I can ‘click to dismiss’ notifications), but it’s not ideal.

A word about stability: on both my main PC and my vintage 2012 HP laptop, Ubuntu has crashed since I performed a fresh install. I can’t say any more than that: there was no common factor I could identify and it could well have been an application common to both installs that was responsible in any case, rather than the OS itself. But I don’t remember OpenSuse ever crashing.

Finally: does it run Oracle? Why, yes it does… though I don’t think I’ve ever seen so many linking errors in one install before (I did the 12.1.0.2 install)! I’ll have to see if I can automate the fixes required, because there are more than a dozen of them! But yes, if you persist, you do get it installed OK:

 

…and the database actually works and returns results when asked nicely:

Ubuntu 16.04 is an “LTS” release -Long-term Support- which means it will be patched and supported until 2021. I barely keep an O/S long enough to make it to month 3, let alone year 5, but it’s a nice concept if you value stability and predictability over the new shiny.

Anyway, it’s now been a week. The main PC is functional; the two laptops are good. I can’t say I love Ubuntu: there are annoyances (chiefly the right-hand placement of the windows controls, I think). But there are pleasures too, and overall I’m enjoying it.

Summing up, then:

Good:

  • ZFS integration
  • Graphics performance
  • Looks
  • Default fonts
  • Horizontal launcher
  • Weird Global and disappearing application menus can be fixed
  • Oracle 12c runs on it

Bad:

  • Window controls unmovable on the left-hand side,
  • Dash icon size: big and un-alterable,
  • The workspace switcher doesn’t switch between virtual desktops… extra clicking is required!
  • Oracle 12c has to be arm-twisted to run on it!
  • Some crashes (not sure if it’s O/S’s fault, though)
  • Relatively poor choice of default wallpapers (fixable by borrowing Xubuntu’s, and others)

Ugly:

  • Slow and flakey installation process
  • The Global Application Menu (which can fortunately be turned off)
  • Notifications are visually intrusive and not very functional

Expanding NAS

zfs01For various historical reasons (basically ToH being tight with the purse-strings!), my two HP servers got populated with different disks at different times. One got a set of four 3TB drives; the other got luckier and got a set of four 4TB drives.

4 x 3 in RAID0 gives you about 12TB of (vulnerable!) storage; 4 x4 in RAID5 gives you about 12TB of (relatively safe!) storage. So the two were comparable, and I set them up in these configurations. The server that does media streaming around the house used RAID0, the server that was the ‘source of truth’ for everything used RAID5, and the RAID5 box regularly replicated itself to the RAID0 box, so the fact that the RAID0 box was vulnerable to a single disk failure wasn’t a concern for overall household data integrity.

But it was a bit cludgy, and I’d have preferred to use two equally-sized RAID5 arrays, if only ToH didn’t mind me splashing out (or “wasting” as it was called) about AUD$1000 on a set of four, fresh 4TB disks.

Colour me surprised, therefore, when I discovered Amazon was offering my preferred Western Digital 4TB Red drives at near-enough half-price! Never being one to look a bargain in the mouth, ToH immediately approved the purchase… and thus put me in a bit of a tricky situation, because things have moved on since the days I ran RAID0 and RAID5 arrays. I upgraded the two servers at Christmas, for example; and switched to using ZFS on Solaris x86 -both servers running raidz (the near-equivalent of RAID5), but of different array sizes because of the different disk sizes.

So my problem now is: how does one migrate a 3TB x 4-disk raidz zpool to being a 4TB x 4-disk zpool, assuming one doesn’t just want to destroy the original zpool, re-create it with the new disks and then re-copy all the data onto the new zpool? Well, here’s how I did it…

As root, check whether the existing zpool is auto-expandable and auto-replaceable, and make it so if not:

zpool get all safedata | grep auto

safedata  autoexpand     off                  local
safedata  autoreplace    off                  default

“Autoexpand” means that if I stick a bunch of 4TB disks in as replacements for a bunch of 3TB disks, the array will automatically see and make use of the extra space. “Autoreplace” means that if I swap out a single old disk for a single new one, will the array automatically begin the process of ‘re-silvering’ the new disk -that is, writing the data which used to be on the old disk back to the new without being manually instructed to do so. Both of these seem to be good ideas, but as the results above show, both features are currently off for my zpool, ‘safedata’. So let me start by fixing that:

zpool set autoexpand=on safedata
zpool set autoreplace=on safedata

I should mention at this point that I nevertheless trigger my replacements manually in what follows, though I do end up relying on the auto-expansion capability, as you’ll discover by the end of the piece.

Next, list the disks which are part of the pool:

zpool status safedata
  pool: safedata
 state: ONLINE
  scan: scrub repaired 0 in 7h21m with 0 errors on Fri May  6 18:33:01 2016

config:

        NAME        STATE     READ WRITE CKSUM
        safedata    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0

errors: No known data errors

So we know the device names from that listing and can see that all devices are online. All you need to do now is to offline one of the disks. I’ll start with the c3t0d0 drive (which I guess to be the one sitting in the left-most drive bay of my server… there’s no way to actually tell for certain with my hardware, unfortunately!):

zpool offline safedata c3t0d0

To confirm what has happened, I re-check the zpool status:

[email protected]:~# zpool status safedata
  pool: safedata
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub repaired 0 in 7h21m with 0 errors on Fri May  6 18:33:01 2016

config:

        NAME        STATE     READ WRITE CKSUM
        safedata    DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0     0
            c3t0d0  OFFLINE      0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0

errors: No known data errors

As expected, the whole pool is now running in DEGRADED status (just like RAID5 would). This isn’t fatal to the zpool as a whole, but isn’t a good thing, either: if I had a disk failure now, I’d lose all my data!

Another unfortunate thing about my HP microservers: the drives aren’t hot-swappable. So though I think I’ve offlined the left-most drive in my array, I can’t just pop that out and see. At this point, therefore, I have to shutdown my server completely and when it’s down, remove drive 1 and reboot. I check my zpool status again and if it still lists it as being ‘degraded’ as before, then we know for certain that c3t0d0 really was the left-most drive in the server. (If it wasn’t, we’d have pulled a good drive out of a degraded array, and at that point the zpool as a whole would be listed as OFFLINE. You could then replace the drive you pulled, reboot once more and re-check the zpool status …and keep going through the drives until you finally pull the correct one).

Assuming you did pull the correct drive, shut the server down again and you can now physically install the new 4TB drive in the correctly-vacated drive slot. Power the server up and issue these commands:

zpool online safedata c3t0d0
zpool replace safedata c3t0d0

Now check the zpool status once more:

[email protected]:~# zpool status safedata       
  pool: safedata
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool 
        will continue to function in a degraded state.
action: Wait for the resilver to complete.
        Run 'zpool status -v' to see device specific details.
  scan: resilver in progress since Fri May 20 10:31:52 2016
    660G scanned out of 6.88T at 3.58G/s, 29m43s to go
    0 resilvered
config:

        NAME              STATE     READ WRITE CKSUM
        safedata          DEGRADED     0     0     0
          raidz1-0        DEGRADED     0     0     0
            replacing-0   DEGRADED     0     0     0
              c3t0d0/old  UNAVAIL      0     0     0
              c3t0d0      DEGRADED     0     0     0  (resilvering)
            c3t1d0        ONLINE       0     0     0
            c3t2d0        ONLINE       0     0     0
            c3t3d0        ONLINE       0     0     0

errors: No known data errors
You have new mail in /var/mail/root

Notice how the new drive has been detected and is being resilvered. You must wait for this resilvering process to complete. You pull another disk out at this point and your zpool is toast. Happily, it gives you an estimate of when it will be finished. Note that there are two parts to the process: scanning the data on the surviving disks -which is what is being mentioned above- and then actually writing the data back to the new disk, which takes a lot longer to complete. In any case, you aren’t left entirely guessing when its work might be done.

So: just keep doing ‘zpool status’ commands to see where it’s got to:

  scan: resilver in progress since Fri May 20 10:31:52 2016
    6.88T scanned
    1.61T resilvered at 364M/s, 93.28% done, 22m10s to go
config:

        NAME              STATE     READ WRITE CKSUM
        safedata          DEGRADED     0     0     0
          raidz1-0        DEGRADED     0     0     0
            replacing-0   DEGRADED     0     0     0
              c3t0d0/old  UNAVAIL      0     0     0
              c3t0d0      DEGRADED     0     0     0  (resilvering)
            c3t1d0        ONLINE       0     0     0
            c3t2d0        ONLINE       0     0     0
            c3t3d0        ONLINE       0     0     0

errors: No known data errors
[email protected]:~# date
Friday, May 20, 2016 04:00:35 PM AEST

When the re-silver has finished, you will see this sort of thing:

[email protected]:~# zpool status safedata
  pool: safedata
 state: ONLINE
  scan: resilvered 1.72T in 5h53m with 0 errors on Fri May 20 16:25:44 2016

config:

        NAME        STATE     READ WRITE CKSUM
        safedata    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0

errors: No known data errors
[email protected]:~# date
Friday, May 20, 2016 04:30:03 PM AEST

Note how the pool is no longer degraded; neither is the c3t0d0 disk… and the old one is no longer even listed. Job done… for that disk, at least.

Once drive #1 has been re-silvered successfully, you can repeat the process for each of the other disks, one at a time, in turn. It will take a while, but eventually, if you do it carefully, you’ll have a newly-sized raidz zpool without having had to destroy and re-create the original.

In my case, each disk took around 5 hours to re-silver. So, for all four disks, it basically took me about a day.

At the end of it, the file system declares itself satisfied:

[email protected]:~$ zpool list
NAME       SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
rpool      111G  24.9G  86.1G  22%  1.00x  ONLINE  -
safedata  14.5T  7.17T  7.33T  49%  1.00x  ONLINE  -

ZFS’s ability to report on physical disk or array size are never entirely transparent to me, but if it’s claiming I’ve got 14.5TB, I assume that’s what 4 x 4TB looks like in “real money” (i.e., accounting for disk manufacturer’s inability to count in base 2, and for similar assorted space overheads). Compare it with the zpool list I got before I started all this, anyway:

[email protected]:~# zpool list
 NAME       SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
 rpool      111G  11.3G  99.7G  10%  1.00x  ONLINE  -
 safedata  10.9T  7.32T  3.55T  67%  1.00x  ONLINE  -

You see how safedata used to report just 10.9TB; now it’s 14.5TB -an increase of about 3.6TB, or roughly the 1 extra TB per disk for four disks that I would hope to have seen! The figures might be a little bit ‘odd’, therefore, in their specific values; but they do generally indicate the zpool has automatically grown to accommodate the new disks supplied.

Which is all good and makes me happy.

Automatic MP3

mp3I rip all my CDs to FLAC: hard disks are capacious and cheap and there’s no point losing audio fidelity for a few tens of megabytes.

But when I take my music on the train of a morning, I don’t much care for audio fidelity: at that point, the fact that my phone only has 64GB of storage matters more. So at this point, converting all those FLAC files to something more compact -like MP3- becomes important.

For many years, therefore, I have laboriously ripped every new CD to FLAC files on disk, copied those FLAC files into a separate mp3 folder and then transcoded them down into fairly low-bit MP3 files there, disposing of the FLAC copies when done. In other words, I end up with 49,000 FLAC tracks in one directory and 49,000 MP3 files in another… and if I remember to do a lot of manual copying and converting, I keep my MP3 folders in sync with their FLAC ‘parents’.

But of course, I often forget to be quite so efficient! I will quite often notice my FLAC taggings are not quite right months after I ripped the CD and thus correct them there… and seldom remember to also go off to the equivalent MP3 folder and correct the ID tags in the same way there, too.

Fundamentally, having two copies of music like this is a bit like having denormalized tables in your database: a lot of repetitive maintenance is required and if you don’t do it perfectly at all times, you start getting data divergence (which is not good!)

Technology to the rescue, therefore!

Linux users have long been able to use “Fuse” (a ‘file system in userspace’) to create ‘virtual file systems’. EncFS, for example, lets you create a file system that transparently encrypts everything you write to it. WikipediaFS creates a file system that displays Wikipedia articles as though they were files.

And mp3fs lets you create a file system that transparently transcodes FLAC files from a source directory into MP3 files sitting in what looks like their own file system. Meaning, you now only have to maintain the FLAC file system’s contents; the equivalent files in the MP3 mirror system are automatically updated next time you access them.

Steps to achieve this nirvana on OpenSuse Leap 42 are as follows:

As root, install the necessary software pre-requisites. This may vary depending on what you’ve already got installed (or not), but I had to do the following:

zypper in libid3tag-devel libmp3lame-devel fuse-devel zypper gcc gcc-c++ flac-devel

Next, obtain the mp3fs software. Unfortunately, there’s no OpenSuse package for this that I could find, so it’s roll-your-sleeves up time and compile from source.

First get your software from here (click the yellow ‘TAR.GZ’ button). Save that somewhere, then type the following commands as yourself:

tar -zxvf mp3fs-0.91.tar.gz
cd mp3fs-0.91

(The specific version number in these commands will change over time, of course: use whatever version number is current at the time you do the software download).

Next (still as yourself):

./configure
make
sudo make install

The last command runs with root privileges to copy files into your /usr/local/bin directory.

Now to test things out. First, we’ll start with a little test run:

sudo mkdir /sourceflacs
sudo mkdir /mp3
sudo chmod 777 /sourceflacs
sudo chmod 777 /mp3

(Security is not important for these purposes, so I don’t mind 777’ing the directories into submission. Your mileage may well vary, of course).

So now, as yourself, copy a FLAC file into the /sourceflacs directory from somewhere:

cp <some flac file> /sourceflacs

Here’s me doing one at a command prompt:

mp3fs01

So you can “ls” that source directory and see a single FLAC, as per the screenshot. Now go and visit the /mp3 directory:

[email protected]:/sourceflacs> cd /mp3
[email protected]:/mp3> ls
[email protected]:/mp3> ls -l
total 0

…Nothing! Not surprisingly: we’ve installed the mp3fs software but we haven’t actually used it yet! So let’s fix that now:

[email protected]:~> cd
[email protected]:~> mp3fs -b 192 /sourceflacs/ /mp3/ -o ro
[email protected]:~> ls /mp3/*
/mp3/03 - Vivace.mp3

So I “cd” first, to make sure I’m sitting in my home directory. Then I invoke the mp3fs executable, specifying to convert my flac files to MP3 files using a bitrate of 192 bits per second (which is reasonable quality). I specify the source and destination directories -and I also make the destination directory read-only. That means that the file system can convert music into it, but I can’t then try editing any of the MP3 files’ tags, for example: in this setup, there’s no point doing so, because the ‘source of truth’ is just the FLAC file in any case.

As you see in the last couple of lines above, once I’ve issued the mp3fs command, I can then list the /mp3 directory and see …something. One file appears there as if by magic: it is, of course, a transcoded version of the flac file I put into the source directory originally.

Actually, although the file name appears in lists of this sort, the file doesn’t physically exist as yet. That only happens when you access the mp3 file, by attempting to play it, for example. (This, incidentally, means you will hear a pause and some stuttering in the background as you first play the file, because all that transcoding activity starts to happen at that point):

mp3fs02

Here you can see me sitting in my “mp3” directory, playing the track in VLC. You see the player is not only playing the file, it has coped with the tagging (it knows that Marcel Poot is the composer for example: look in its header bar. Look also at the directory and file name: it’s not getting that information from anywhere other than the ID3 tags in the music file). The album art has made the journey across, too, which is a bonus.

Just to ram home the point: I’ve just made a transparent-encoding-on-the-fly copy of a FLAC file. If I alter the original (say I change its composer/artist tags) and stop and re-play the MP3 file, the MP3 version will automatically pick up those changes too. So here is me altering the tags in the /sourceflacs folder, using my new-found-favourite ID3 tagging software, “puddletag”:

mp3fs03

With those changes saved, let me replay the version of the file in the /mp3 folder:

mp3fs04

And look who is listed as the work’s composer in VLC’s header bar now!

So this works very well for a single file, as a simple example. But in my case, I want to mount my entire collection of flacs as a source for the mp3fs magic -and I want that done automatically at boot time, not just by manually issuing an ‘mp3fs’ command.

So first, let’s get rid of the test case we’ve worked with so far. That is, I want to empty the contents of the ‘/mp3’ directory. That’s done by un-mounting the mp3fs file system, which has to be done as root (and so long as no part of any file stored within the /mp3 directory is being played or otherwise accessed):

sudo umount /mp3

Next, as root, you need to edit your /etc/fstab directory and make a one-line addition to it, like so:

sudo nano /etc/fstab
mp3fs#/sourceflacs /mp3 fuse allow_other,ro,bitrate=192 0 0

The “allow_other” option permits users other than root to access the mounted file system, so long as the directory permissions are OK (which they are, of course, in my case). The tail-end of my fstab file therefore ends up looking like this:

mp3fs05

Once that addition has been saved, the new file system can be manually mounted with a simple sudo mount /mp3 …or you could just reboot your server! (But see the update at the end of this blog piece).

Bear in mind that I can mp3fs mount my entire 1TB collection of flacs in no time at all using this mechanism: the files don’t exist until you play them (which means you don’t need much disk space to do this in, either!) What’s more, once you’ve played them, they cease to exist! Oh, they will appear to exist in directory listings and so on, but appearances in this case are deceptive.

So, say for example that I mounted my entire collection of ~49,000 flac files with mp3fs. An exact number of MP3 files will then immediately appear in my /mp3 folder: the file system doesn’t transcode the files in the background or anything, so it takes practically no time at all for them all to seem to exist.

Now suppose I want to take an off-site backup of my MP3 files. So I do something like:

cp /mp3/*.mp3 /media/usb

MP3 files being relatively small, this shouldn’t take much time, right? Wrong. The copy command causes each file to be transcoded and instantiated in real time by the mp3 file system before the file is passed to the USB drive. So your copy command will require however long it takes to convert all your files from flac to mp3, plus the time it takes to copy the converted files.

And when all that is done, you will have proper, permanent MP3 files on your USB drive (which will genuinely need lots of disk space to house them all)… and the files in your /mp3 directory will still not really exist! Play one of them now and it will have to be generated by real-time transcoding all over again.

In other words, this really is a virtual file system and nothing really exists on it except when you are actually accessing it. And once your access (be it playing it, copying it etc) has finished, so the file ceases to exist once more. So fair warning then: use of this sort of thing on your desktop would imply you need a fairly good disk I/O subsystem and a reasonable CPU, because if you access the “mp3” files very often, your flacs will be read and converted a lot, too.

But that’s not a problem for me: I don’t need my MP3 files to persist on my desktop PC. I just need them to appear automatically when I need them, for copying to my phone: so long as the files are permanent on that device, that’s fine. So for me, mp3fs is a truly nice fix for a problem that has annoyed me for years! Would that I had known of it earlier (and I should have: its first code commits were made ten years ago!!)

Better late than never, I guess.

UPDATE: everything I wrote above is true for normal file systems (like ext4 or xfs). Unfortunately, I forgot that my source of flacs is actually stored on a ZFS file system. This makes a difference, because ZFS on OpenSuse doesn’t currently do automatic mounts at boot-time. So if your fstab says to auto-mount an mp3fs file system that is dependent on a ZFS system that itself is not auto-mounted, you’ll be in trouble (like: “can’t boot this machine, so dropping to an emergency command line environment” trouble!)

So if your source of flacs isn’t automounted via /etc/fstab, you cannot put an entry for mp3fs in your /etc/fstab either. Instead, you have to mount things a different way.

On my OpenSuse box, I mount my ZFS file system via an init script (as described in principle in this article). Specifically, I have a directory called /etc/init.d/mountzfs.d which contains a script called “start” which simply says to run a /root/mountmusic.sh shell script. That script in turn contains these commands:

#!/bin/bash
sudo modprobe zfs
sudo zpool import safedata

So that’s how I get my zfs volumes to appear to auto-mount at every reboot.

It’s then simply a question of adding the manual mp3fs command to that script so that it not only mounts the source flac data but also goes on to mount my mp3fs virtual file system, too. Here’s my revised mountmusic.sh script, therefore:

#!/bin/bash
sudo modprobe zfs
sudo zpool import safedata

mp3fs -b 192 /safedata/flac/ /safedata/mp3/ -o allow_other,ro

If, like me, you forgot this minor glitch, use the emergency command line environment to edit the fstab file and comment out all mention of mp3fs mounts and you’ll be able to boot properly once more. Once your PC boots correctly, editing the ZFS auto-mount script in this way becomes trivially easy.

 

 

Before… and After

It took a while, but five days ahead of schedule, Amazon delivered my second PCI Express 2-port SATA expansion card. That allowed me to switch my server #1 back to AHCI mode instead of the “Legacy SATA” mode it had been in since I first built it.

The question left hang from my earlier blog piece about this issue was whether being in a different SATA mode would make an appreciable difference to the speed with which Server #1 could grab data from Server #2.

By way of reminder, here is the network speed I was getting before:

And here’s what I’m getting after the switch to AHCI mode:

I make that a 72% improvement in network speed -because it’s not being throttled by such poor disk speeds anymore.

Similarly, the best speed I was getting for a scrub before was:

48.4G scanned out of 7.32T at 169M/s, 12h34m to go

But after:

14.2G scanned out of 4.14T at 324M/s, 3h43m to go

…a 92% improvement. And that was measured whilst the ‘missing’ 3.2TB of data was being re-copied back to server 1, so the disks were pretty busy elsewhere!

So the short version of this saga is: Solaris works well on the HP Generation 8 microservers, provided you run them in AHCI mode (not legacy or RAID modes). And doing that will require you obtain an el-cheapo PCIe SATA card to allow you to boot from a fifth disk, leaving the 4 ‘proper’ disk slots available for pure data storage purposes.

It took a while… but I got there in the end!