Asquith 2.0

I’ve decided: There will be no Asquith 2.0 that runs on Red Hat/CentOS 7.

There are a lot of stumbling blocks, some of which I’ve documented here recently -including things like iSCSI target configurations no longer being easily scriptable, the use of systemd and the use of dynamic names for network devices. No doubt, all of these problems will be resolved over time by the upstream developers, but they currently make it practically impossible to construct a highly-automated, self-building Asquith framework. (Salisbury, needing only NFS, is a much better proposition, but even there the network device naming issue presents automation difficulties).

Since Red Hat 6.x is supported until 2020, I’ll pass on Red Hat 7 and its assorted clones. I rather imagine quite a lot of real-life enterprises might do the same!

Harper and Owen

Somewhat against my better judgment, we spent Saturday visiting a local (translation: an hour up the freeway!) rescued animal shelter. What a miserable place it was! The constant howling from the dog section was quite troubling (I really don’t understand people who keep dogs. They are quite the most anti-social statement a neighbour could make, I think). But the palpable sense of doom experienced in the cat section was worse.

Don’t get me wrong: Renbury Farm is a fine place doing excellent work, and the people there seemed incredibly caring and knowledgeable. But so many of the cats were hiding at the back of their cages, scared out of their wits; some were practicaly feral and accordingly hissing and spitting at passers-by. All are under imminent sentence of death, basically: they have a week to be claimed or ‘adopted’ and then their time is up. Hence all the “Save by 3pm Monday” messages on the cat pages of their website (update: site now replaced by a Facebook page that isn’t quite as time-alarming!).

Their resources are limited, of course, so they have no choice but to take them in, ship ’em out and cull the ones that can’t make the journey in time. But it’s sad. And made all the worse, because apparently clueless humans are the cause of all the grief in the first place.

Anyway, much as we’d have liked to be able to save a good four or five of them, we decided we could only realistically adopt two of them. Both males, both about 2 years old. One is to be called Owen:

Owen is named after Owen Wingrave, the eponymous hero of Benjamin Britten’s 1971 opera. He’s also named after Wilfrid Owen, since his official birthday is listed as August 4th and in the centenary of the outbreak of World War I, it seemed sort-of appropriate. Anyway, I like the name!

The other is Harper:

My sister tells me that people will think we named Harper after David Beckham’s daughter, someone of whose existence I was not previously aware. Folks that know me better will realise it’s an obvious allusion to the fact that it’s a sin to kill a mockingbird (or to be unkind to cats). Which is apparently where David got the name from, too, so we’re cool.

Both are currently at Renbury’s vets being de-wormed, vaccinated and (sorry, boys!) de-sexed. We pick them up tomorrow. I am unsure whether it is wise to replace long-time companion pets so soon after their demise, but I am looking forward to bringing them home nonetheless. Time will tell if the boys prove worthy successors to Lucretia and Gracie.

Scientific Revelations

Continuing my recent theme of “CentOS is now not very useful”, I thought I should check up on how Scientific Linux version 7 was getting on… and got a bit of a surprise.

Scientific Linux was a joint project of CERN (the European atom-smashers) and Fermilab (the American equivalents) and dates back to the days of Red Hat Enterprise Linux 3 in 2004. So it’s venerable -and supported and developed by two prestigious and highly-trustworthy organisations. In the past, that’s been important -because the lack of transparency shown by the CentOS developers at times meant that there was precious little trust to be had in that particular distro!

In January this year, however, Red Hat announced it was ‘going into partnership’ with CentOS: it hired some of their lead developers, for example. If it’s possible to buy-out an open source project, Red Hat can fairly be said to have bought out CentOS. This announcement caused the CERN part of the Scientific Linux team to decide not to bother developing their own SL7 distro: instead, they said, they would adopt CentOS 7 and seek to become a ‘CentOS Special Interest Group’, giving them the chance to customise the basic CentOS installation.

Only the Fermilab developers decided to pursue their own, independent Scientific Linux distro into its version 7.x incarnation.

I’m glad someone is still pursuing a dedicated Scientific Linux distro: the competition and point of comparison with CentOS has been quite important in the past (especially when the CentOS team appeared to have completely lost the plot, around the time that version 6.0 was released). If all Enterprise Linux distros were just to be respins of CentOS (which is now, remember, essentially a not-paid-for subsidiary of Red Hat itself), we would be much the poorer for it. I don’t fancy a world in which I only have a “choice” between Red Hat and Oracle, for example.

Happily, Fermilab have indeed pushed on and a release of their version 7.0 is nearly upon us: the second beta was made available last Friday, for example. Here’s to their continued development efforts: we need them to succeed and to keep on wanting to succeed.

CentOS 7 – Not Recommended

I’ve been poking around with CentOS 7, Red Hat 7 and Oracle Enterprise Linux 7 extensively in the past couple of weeks, in the hope of producing a 7-compliant version of Asquith. For the sake of the rest of this post, lets agree to call all those distros, generically, Enterprise Linux 7.x

It’s been quite a ride, because an awful lot has been changed in the transition from Enterprise Linux 6.x to 7.x. I’ll list just some of the differences that have specifically tripped me up here:

  • How you invoke a Kickstart installation in the first place has changed
  • How you do firewalling has changed (firewalld not iptables)
  • The packages and package groups has changed
  • The way you configure iSCSI targets has changed, dramatically, and uses an interactive shell that isn’t suitable for scripting and doesn’t work with Kickstart
  • The way you disable and enable, stop and start services has changed (systemd v. init scripts)
  • The way network devices are named is now “intelligent”… and completely borks Kickstart

I’ll just explain a little more about that last point, by the way, since I’ve not mentioned it at all in any previous blogs. The gist of it is that you probably know and love your network interfaces as things like “eth0” and “eth1”, and have done for years. But they aren’t called that any more. Oh no. Instead, you get names such as “enp0s3” and “eno16777736” …and (this is the particularly cunning bit): you get different names depending on what your hardware and your BIOS is capable of.

The idea behind the change is logical and admirable in and of itself: in a server with two Ethernet cards, you were never entirely sure which one would pick up the “eth0” designation and which the “eth0”. Whereas now, the names are bus/slot dependent and are thus assigned deterministically: the card in slot 3 gets the ‘s3’ name and the one in slot 4 gets an ‘s4’ name. Simple, although you won’t know what your interface is going to be called until after the installation has itemised all your hardware and assigned the appropriate names.

The trouble is that in Kickstart, we used to do this sort of thing:

network --device eth0 --bootproto static --ip --netmask --nameserver --hostname alpher.dizwell.home
network --device eth1 --bootproto static --ip --netmask --nameserver  hostname alpher-priv

That is to say: assign one set of network attributes to an interface called “eth0” and another to one called “eth1” …and do those assignments before the installation has completed.

So how do you write equivalent lines for an Enterprise Linux 7 Kickstart script when don’t know whether your interfaces are going to be called “enp0s3” or “eno16777736” or something completely different until after the installation has finished? You can’t. It’s just impossible to write one Kickstart script to run on any hardware now, because you won’t know what device names are ahead of time.

A case in point: my laptop and my home desktop PC, running a virtual machine in VMware Workstation, both produce Linux guests that end up with a network interface called “eno1677736”, but my work PC (also running VMware) produces guests that have a “enp0s3” network interface. One Kickstart script cannot do duty for both environments… and I have no idea what other variants my readers and would-be Asquith users might end up with, so I can’t even start taking them into account!

It’s a mess. In plain words: Enterprise Linux 7 breaks Kickstart installations.

I’ve had to work around it for now by reducing the Kickstart network line down to:

network --hostname=asquith.dizwell.home

…which doesn’t attempt to configure networking interfaces at all. Later on in my Kickstart script, in the %post section, I then cheat like mad and run this:

for f in `ls /sys/class/net`; do
  if [[ $f != "lo" ]]; then 
    cat > /etc/sysconfig/network-scripts/ifcfg-$f << EOF
NAME="System $f"

…which simply writes a network configuration file for any non-loopback interface it finds listed in the /sys/class/net directory. This is taking place after the installation has all-but completed, so by that stage the finished interface names should be available. As that stands, it will write the same configuration for both interfaces in a 2-interface server, which is obviously not right… but a little bit of bash if-then-else’ing should see that right. For Asquith itself, which only has one network interface to worry about, this code as written will work no matter what interface names your installation decides to bestow upon your guest.

But it’s not “right” and it’s certainly not elegant, and the fact can’t be dodged that in their eagerness to embrace “meaningful network interface names”, the Enterprise Linux developers broke Kickstart. Hopefully, they will un-break it before too long.

Anyway… all these changes I’ve been whinging about of late apply equally well to all three variants of Enterprise Linux that I’ve been working with. It doesn’t matter whether you use CentOS, RHEL or OEL: Kickstart (for example) will struggle to configure your network interfaces correctly in all of them.

I do, however, have a special word of opprobrium to hurl uniquely in CentOS’s direction: it’s the only one of the three distros that decided it was more important to have a word processor on your Enterprise Linux server than a working Oracle database.

Let me explain: When you register for the Red Hat trial and download the 3.4 GB Red Hat 7 ISO; or when you download the Oracle Enterprise Linux 4GB ‘V46135-01’ ISO…. in both cases you end up with a single DVD image which contains a mix of 32-bit and 64-bit libraries/packages. As you know, Oracle’s Linux installs still require a mix of 32-bit and 64-bit packages to work properly (for example, you need glibc-x86_64 and glibc-i686 before things will compile correctly during the ‘linking phase’ of the Oracle database installation). So Red Hat and OEL both provide distro installation media which can satisfy those requirements.

But if you download the 4GB CentOS 7 installation DVD, you get pure, 64-bit only packages. No .i686 software exists at all, and thus no Oracle software installations are possible with it. I asked on the CentOS forums why they decided to package things up quite differently than their upstream vendor (i.e., Red Hat) did and the only explanation someone offered was that “to make room for the LibreOffice software, they had to ditch the i686 libraries”. I’m not sure if that’s so (a DVD ISO can be 4.7GB, so there’s room for 700MB of extras on the CentOS DVD even as it stands), but if it were true, it’s a weird choice: we package our Enterprise Linux distro so that you can run a word processor, but not an Enterprise-class database. You figure the logic of that, because I can’t see any in it.

You can, of course, install Oracle database software on CentOS by the “simple” expedients of either (a) connecting your server directly to the Internet and downloading the relevant 32-bit packages with yum; or (b) downloading the CentOS “Everything” ISO, instead of the plain-vanilla “DVD ISO”. The Everything ISO is 6.7GB in size and does include the i686 software packages you’ll need. But that means it’s nearly 3GB bigger than OEL or RHEL’s Oracle-ready equivalents.

I shall be interested to see how Scientific Linux do their packaging when the time comes (they are currently stuck at version 6.5, so I don’t know when or if SL7.0 will be making an appearance).

In the meantime, I shall have no choice but to strongly recommend NOT using CentOS 7 as a platform for Oracle databases. I’ll be switching all my development work to OEL 7.x, which is Oracle-database-ready AND can be downloaded and updated for free. CentOS just seems too weirdly and obtusely different from the other Enterprise Linux distros to be worth bothering with at the moment.

Updated to add: This isn’t the only point at which CentOS diverges in annoying ways from Red Hat or Oracle’s treatment of what is supposed to be essentially the same distro: try doing an lsb_release -r to see what version your distro reports itself to be. Red Hat reports 7.0. OEL reports 7.0. CentOS, however, decides it will be clever and report 7.0.1406. Version number reporting is important to Asquith, because it determines where you’ll fetch your software from when building client servers. Having one distro decide to be different from all the others is therefore, frankly, rather annoying!


I don’t often share music stuff on this blog these days, largely because what I listen to is generally considered a “minority interest” if I ever discuss it with anyone! Additionally, I am pretty set in my musical ways, as will show you: Britten and Bach head-and-shoulders above anything else, then a fair chunk of 18th Century stuff and another fair chunk of 20th Century stuff with not much in between, with a strange exception made for Verdi. So I don’t frequently come across stuff I am bowled over by that I think might merit a blog mention.

On the other hand, in my college days, and my first few years in Australia, I sang in Chapel, Church and Cathedral choirs where the likes of Tallis, Byrd, Desprez, Vittoria and Lassus were fairly standard stuff, so if you hunt through my preferences long enough, you’ll see them pop up as semi-regular listening too.

But Antoine Brumel has never previously been someone I’ve listened to, ever. To be honest, he’s not someone I’ve ever actually heard of before, probably because he was French! But I have just heard his ‘Earthquake Mass’ for the first time, and have been amazed by it. It’s a rather peculiar piece in that, over the span of the entire work, it sounds like only two or three chord progressions take place: it’s got little apparent harmonic subtlety or progression. But within that wall of basic harmony are amazing rhythmic effects that constantly engage the ear. I can’t stop playing it of late -and I don’t say that very often about pre-Reformation French choral music!!

I think copyright fair use doctrine means I can share the last minute or two of the Kyrie from the mass here, so you can get a sense of what this ‘wall of sound’ is like:

And if that impresses you as much as it has me, then you can buy the entire thing from here.