I rip all my CDs to FLAC: hard disks are capacious and cheap and there’s no point losing audio fidelity for a few tens of megabytes.
But when I take my music on the train of a morning, I don’t much care for audio fidelity: at that point, the fact that my phone only has 64GB of storage matters more. So at this point, converting all those FLAC files to something more compact -like MP3- becomes important.
For many years, therefore, I have laboriously ripped every new CD to FLAC files on disk, copied those FLAC files into a separate mp3 folder and then transcoded them down into fairly low-bit MP3 files there, disposing of the FLAC copies when done. In other words, I end up with 49,000 FLAC tracks in one directory and 49,000 MP3 files in another… and if I remember to do a lot of manual copying and converting, I keep my MP3 folders in sync with their FLAC ‘parents’.
But of course, I often forget to be quite so efficient! I will quite often notice my FLAC taggings are not quite right months after I ripped the CD and thus correct them there… and seldom remember to also go off to the equivalent MP3 folder and correct the ID tags in the same way there, too.
Fundamentally, having two copies of music like this is a bit like having denormalized tables in your database: a lot of repetitive maintenance is required and if you don’t do it perfectly at all times, you start getting data divergence (which is not good!)
Technology to the rescue, therefore!
Linux users have long been able to use “Fuse” (a ‘file system in userspace’) to create ‘virtual file systems’. EncFS, for example, lets you create a file system that transparently encrypts everything you write to it. WikipediaFS creates a file system that displays Wikipedia articles as though they were files.
And mp3fs lets you create a file system that transparently transcodes FLAC files from a source directory into MP3 files sitting in what looks like their own file system. Meaning, you now only have to maintain the FLAC file system’s contents; the equivalent files in the MP3 mirror system are automatically updated next time you access them.
Steps to achieve this nirvana on OpenSuse Leap 42 are as follows:
As root, install the necessary software pre-requisites. This may vary depending on what you’ve already got installed (or not), but I had to do the following:
zypper in libid3tag-devel libmp3lame-devel fuse-devel zypper gcc gcc-c++ flac-devel
Next, obtain the mp3fs software. Unfortunately, there’s no OpenSuse package for this that I could find, so it’s roll-your-sleeves up time and compile from source.
First get your software from here (click the yellow ‘TAR.GZ’ button). Save that somewhere, then type the following commands as yourself:
tar -zxvf mp3fs-0.91.tar.gz
(The specific version number in these commands will change over time, of course: use whatever version number is current at the time you do the software download).
Next (still as yourself):
sudo make install
The last command runs with root privileges to copy files into your /usr/local/bin directory.
Now to test things out. First, we’ll start with a little test run:
sudo mkdir /sourceflacs
sudo mkdir /mp3
sudo chmod 777 /sourceflacs
sudo chmod 777 /mp3
(Security is not important for these purposes, so I don’t mind 777’ing the directories into submission. Your mileage may well vary, of course).
So now, as yourself, copy a FLAC file into the /sourceflacs directory from somewhere:
cp <some flac file> /sourceflacs
Here’s me doing one at a command prompt:
So you can “ls” that source directory and see a single FLAC, as per the screenshot. Now go and visit the /mp3 directory:
[email protected]:/sourceflacs> cd /mp3
[email protected]:/mp3> ls
[email protected]:/mp3> ls -l
…Nothing! Not surprisingly: we’ve installed the mp3fs software but we haven’t actually used it yet! So let’s fix that now:
[email protected]:~> cd
[email protected]:~> mp3fs -b 192 /sourceflacs/ /mp3/ -o ro
[email protected]:~> ls /mp3/*
/mp3/03 - Vivace.mp3
So I “cd” first, to make sure I’m sitting in my home directory. Then I invoke the mp3fs executable, specifying to convert my flac files to MP3 files using a bitrate of 192 bits per second (which is reasonable quality). I specify the source and destination directories -and I also make the destination directory read-only. That means that the file system can convert music into it, but I can’t then try editing any of the MP3 files’ tags, for example: in this setup, there’s no point doing so, because the ‘source of truth’ is just the FLAC file in any case.
As you see in the last couple of lines above, once I’ve issued the mp3fs command, I can then list the /mp3 directory and see …something. One file appears there as if by magic: it is, of course, a transcoded version of the flac file I put into the source directory originally.
Actually, although the file name appears in lists of this sort, the file doesn’t physically exist as yet. That only happens when you access the mp3 file, by attempting to play it, for example. (This, incidentally, means you will hear a pause and some stuttering in the background as you first play the file, because all that transcoding activity starts to happen at that point):
Here you can see me sitting in my “mp3” directory, playing the track in VLC. You see the player is not only playing the file, it has coped with the tagging (it knows that Marcel Poot is the composer for example: look in its header bar. Look also at the directory and file name: it’s not getting that information from anywhere other than the ID3 tags in the music file). The album art has made the journey across, too, which is a bonus.
Just to ram home the point: I’ve just made a transparent-encoding-on-the-fly copy of a FLAC file. If I alter the original (say I change its composer/artist tags) and stop and re-play the MP3 file, the MP3 version will automatically pick up those changes too. So here is me altering the tags in the /sourceflacs folder, using my new-found-favourite ID3 tagging software, “puddletag”:
With those changes saved, let me replay the version of the file in the /mp3 folder:
And look who is listed as the work’s composer in VLC’s header bar now!
So this works very well for a single file, as a simple example. But in my case, I want to mount my entire collection of flacs as a source for the mp3fs magic -and I want that done automatically at boot time, not just by manually issuing an ‘mp3fs’ command.
So first, let’s get rid of the test case we’ve worked with so far. That is, I want to empty the contents of the ‘/mp3’ directory. That’s done by un-mounting the mp3fs file system, which has to be done as root (and so long as no part of any file stored within the /mp3 directory is being played or otherwise accessed):
sudo umount /mp3
Next, as root, you need to edit your /etc/fstab directory and make a one-line addition to it, like so:
sudo nano /etc/fstab
mp3fs#/sourceflacs /mp3 fuse allow_other,ro,bitrate=192 0 0
The “allow_other” option permits users other than root to access the mounted file system, so long as the directory permissions are OK (which they are, of course, in my case). The tail-end of my fstab file therefore ends up looking like this:
Once that addition has been saved, the new file system can be manually mounted with a simple sudo mount /mp3 …or you could just reboot your server! (But see the update at the end of this blog piece).
Bear in mind that I can mp3fs mount my entire 1TB collection of flacs in no time at all using this mechanism: the files don’t exist until you play them (which means you don’t need much disk space to do this in, either!) What’s more, once you’ve played them, they cease to exist! Oh, they will appear to exist in directory listings and so on, but appearances in this case are deceptive.
So, say for example that I mounted my entire collection of ~49,000 flac files with mp3fs. An exact number of MP3 files will then immediately appear in my /mp3 folder: the file system doesn’t transcode the files in the background or anything, so it takes practically no time at all for them all to seem to exist.
Now suppose I want to take an off-site backup of my MP3 files. So I do something like:
cp /mp3/*.mp3 /media/usb
MP3 files being relatively small, this shouldn’t take much time, right? Wrong. The copy command causes each file to be transcoded and instantiated in real time by the mp3 file system before the file is passed to the USB drive. So your copy command will require however long it takes to convert all your files from flac to mp3, plus the time it takes to copy the converted files.
And when all that is done, you will have proper, permanent MP3 files on your USB drive (which will genuinely need lots of disk space to house them all)… and the files in your /mp3 directory will still not really exist! Play one of them now and it will have to be generated by real-time transcoding all over again.
In other words, this really is a virtual file system and nothing really exists on it except when you are actually accessing it. And once your access (be it playing it, copying it etc) has finished, so the file ceases to exist once more. So fair warning then: use of this sort of thing on your desktop would imply you need a fairly good disk I/O subsystem and a reasonable CPU, because if you access the “mp3” files very often, your flacs will be read and converted a lot, too.
But that’s not a problem for me: I don’t need my MP3 files to persist on my desktop PC. I just need them to appear automatically when I need them, for copying to my phone: so long as the files are permanent on that device, that’s fine. So for me, mp3fs is a truly nice fix for a problem that has annoyed me for years! Would that I had known of it earlier (and I should have: its first code commits were made ten years ago!!)
Better late than never, I guess.
UPDATE: everything I wrote above is true for normal file systems (like ext4 or xfs). Unfortunately, I forgot that my source of flacs is actually stored on a ZFS file system. This makes a difference, because ZFS on OpenSuse doesn’t currently do automatic mounts at boot-time. So if your fstab says to auto-mount an mp3fs file system that is dependent on a ZFS system that itself is not auto-mounted, you’ll be in trouble (like: “can’t boot this machine, so dropping to an emergency command line environment” trouble!)
So if your source of flacs isn’t automounted via /etc/fstab, you cannot put an entry for mp3fs in your /etc/fstab either. Instead, you have to mount things a different way.
On my OpenSuse box, I mount my ZFS file system via an init script (as described in principle in this article). Specifically, I have a directory called /etc/init.d/mountzfs.d which contains a script called “start” which simply says to run a /root/mountmusic.sh shell script. That script in turn contains these commands:
sudo modprobe zfs
sudo zpool import safedata
So that’s how I get my zfs volumes to appear to auto-mount at every reboot.
It’s then simply a question of adding the manual mp3fs command to that script so that it not only mounts the source flac data but also goes on to mount my mp3fs virtual file system, too. Here’s my revised mountmusic.sh script, therefore:
sudo modprobe zfs
sudo zpool import safedata
mp3fs -b 192 /safedata/flac/ /safedata/mp3/ -o allow_other,ro
If, like me, you forgot this minor glitch, use the emergency command line environment to edit the fstab file and comment out all mention of mp3fs mounts and you’ll be able to boot properly once more. Once your PC boots correctly, editing the ZFS auto-mount script in this way becomes trivially easy.