Fedora 27

Just as I get Atlas working on Fedora 26, so they go and release a brand new version of it: version 27 was released on November 14th.

There is a ghastly new look (I suggest changing the default jellyfish wallpaper the moment you first log on!), but there is otherwise not a lot listed as being new in the release notes, apart from an upgrade to the latest release of LibreOffice.

Mostly underwhelming, in other words; but still just about the most functional distro out there (imho, naturally). It remains my desktop of choice, anyway. Unfortunately, I can’t actually upgrade my main desktop to version 27 because when I do, I get told:

Problem: package zfs-dkms-0.7.3-1.fc26.noarch requires /bin/ksh, but none of the providers can be installed
 - ksh-20120801-35.fc26.x86_64 does not belong to a distupgrade repository
 - problem with installed package zfs-dkms-0.7.3-1.fc26.noarch

As far as I have been able to tell, this is caused by the ZFS on Linux project not having Fedora 27-compatible repositories ready -and, allegedly, they won’t have one until mid-December. So any Fedora 26 users who have committed to using ZFS for their important data won’t be able to upgrade for a few weeks. Fingers crossed for the December timeline.

My laptop doesn’t use ZFS, though, and was able to upgrade without drama.

I’ve also re-jigged Atlas, so Oracle 12c will run on Fedora 27 without too much fuss (the same need to do a software-only install that plagued version 26 also affects version 27, but the workaround previously described works for the new version of Fedora perfectly well):


Atlas and Solaris 11.3

A reader asked an extremely sensible question on hearing that Atlas has been made to work on OpenIndiana: does it therefore work on “proper” Solaris too?

The short answer to that question is “no”, but I’ll take the opportunity to explain why that is so here.

To run Atlas on any one of 20 different operating systems and/or Linux distros, you first obtain the atlas shell script with a command such as:

wget https://bit.do/dizatlas -O atlas.sh

Wget is an ancient (well, 1996… but that’s ancient in Internet years!) utility that simply fetches data from a web server. The web server in question is a little obfuscated by that short-form “bit.do” URL shown in the above command -but it’s actually my own web server, based in Paris, that hosts dizwell, diznix and some other websites I dabble with from time to time. As I say, wget has zero problem fetching the atlas script contents when told to do so on 20 different distros and OSes.

But, uniquely, on proper Solaris (in this case, 11.3), this happens:

The important bit there is “sslv3 alert handshake failure“: it means that Solaris’ version of wget is built to use the sslv3 protocol, and my web server doesn’t support that. Quite what the version of wget used by OpenIndiana, Ubuntu, Red Hat, etc etc ad infinitum is using to connect to my web server, I really have no idea. But whatever those other wget versions are using, it works happily! Only Solaris uses a wget configured in a way that doesn’t.

Now I respect Solaris’ “bullet-proof” nature, so I rather suspect that it’s my web server configuration that is at fault here, rather than Solaris. But I’m not re-configuring my entire server just because one O/S can’t handle it!

Anyway, Solaris’ inability to fetch stuff from my web server means Atlas falls at the first hurdle: you can’t even download the script to it in the first place! But suppose you persevered and somehow managed to download the relevant file to a Solaris machine via a web browser or some other technique… surely it would work then? Sort of… except that you would then fall at the second hurdle -which is that the atlas script expects to be able to download -using wget- a second script. Since that will fail too, for the same sslv3 reason described above, the entire thing collapses in a heap without configuring a thing on your Solaris box! 🙁

This is annoying!

So I resolved the matter by the simple expedient of storing a copy of the atlas script at Dropbox: Solaris’ version of wget is happy to negotiate a suitable connection with their servers, so a wget from there works fine. Similarly, if I rejig things slightly, the second script download works. And Lo! It then becomes trivially easy to run Atlas on Solaris 11.3 after all. Once you do, Oracle 12c R1 and R2 both install without incident or error.

So the rather longer answer to my original reader’s question is: Atlas didn’t run on proper Solaris, for abstruse technical reasons to do with encryption protocols, secret handshakes and the like… but now it does:

Obviously, I will alter the Atlas documentation again to take account of this new capability (which may take a day or two), but the quick instructions are:

wget https://bit.do/dizatlas2 -O atlas.sh
chmod +x atlas.sh

Note the use of dizatlas2 in that first wget command. That forces wget to talk to the Dropbox copy of Atlas, not the original as stored on my own server. I dislike needing to have copies of things in two entirely independent locations like this (it makes synchronizing the two repositories more difficult than it should be), but it’s the quickest solution for now: since you’ve pointed to the Dropbox repository, the wget command will succeed in fetching the file -and the script will similarly succeed in fetching the second file it needs to do its configuration work.

No other OS or distro needs -or indeed should– use this dizatlas2 respository: it’s only ever useful and required for Solaris 11.

OpenIndiana “Hipster”

The future does not look especially great for Solaris, given Oracle Corporation’s seeming reluctance to hold on to its principle sources of Solaris development expertise.

However, OpenIndiana (which is based on the forked, open source Solaris 10) continues to be fitfully developed and refreshed by its loyal band of devotee developers: a brand new release hit the download servers at the tail-end of October (and is thus called the 1710 release of the Hipster fast development branch of OpenIndiana).

It is probably slightly odd to spend any time worrying about whether Oracle 12c runs on this new version of OpenIndiana, given its niche appeal and the imminent demise of the ‘real thing’ it’s based on. But I did anyway… and I can therefore report that both 12cR1 and 12cR2 install without drama.

12c Release 2 complains of a missing package, called ‘assembler-0.5’, but this warning can be ignored and the installation will continue to a successful conclusion anyway. There are no warnings about anything with 12c Release 1, I’m happy to say.

To make things easier, I’ve re-jigged Atlas to work to get these installations automated. Atlas was always promised to be the ‘universal’ script to get Oracle installed on ‘*nix’, so getting it to work for ‘proper’ Unix as well as for Linux is, if anything, long overdue.

I’ll be updating the Atlas documentation to reflect its new-found OpenIndiana-capabilities in the course of the next day or so. In the meantime, the short version is simply to build a Hipster VM with at least 5120MB of virtual RAM, then:

wget https://bit.do/dizatlas -O atlas.sh
chmod +x atlas.sh

After that, follow the prompts to completion (including a reboot). Then download the Solaris X86 flavour of one of the Oracle 12c database versions, unzip the download(s) and then invoke the Oracle installer by the usual database/runInstaller command.

Red Hat 7.4

Red Hat has just released the latest update to its version 7 Enterprise Linux, so we are now up to RHEL 7.4.

Getting hold of it is, of course, rather more hard work than it would be for most distros, since Red Hat charge a pretty penny for the real thing! However, signing up as a Red Hat developer can be done entirely free of charge… and getting your mitts on it after that is a piece of cake (and costs precisely nothing at all 🙂 )

On to the obvious question: does Atlas work to make Oracle 12cR1 and 12cR2 installations on this new Red Hat version as simple as it did for earlier ones? Absolutely:

The article describing how to do it still holds true without any alteration or update being necessary, too.

What were the odds?

In modernising Churchill to work for Oracle 12c and the latest 6.x releases of RHCSL, I’ve encountered a bizarre bug (#19476913 if you’re able to check up on it), whereby startup of the cluster stack on a remote node fails if its hostname is longer than (or equal to) the hostname of the local node.

That is, if you are running the Grid Infrastructure installer from Alpher (6 characters) and pushing to Bethe (5 characters) then the CRS starts on Bethe just fine: local 6 is greater than remote 5. But if you are running the GI installer on Gamow (5 characters) and pushing to Dalton (6 characters) then the installer’s attempt to restart the CRS on Dalton will fail, since now local 5 is less than remote 6. Alpher/Bethe managed to dodge this bullet, of course -but only by pure luck.

The symptoms are that during the installation of Grid Infrastructure, all works well until the root scripts are run, at which point (and after a long wait), this pops up:

Poke around in the [Details] of that dialog and you’ll see this:

CRS-2676: Start of 'ora.cssdmonitor' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cssd' on 'dalton' 
CRS-2672: Attempting to start 'ora.diskmon' on 'dalton' 
CRS-2676: Start of 'ora.diskmon' on 'dalton' succeeded 
CRS-2676: Start of 'ora.cssd' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dalton' 
CRS-2672: Attempting to start 'ora.ctssd' on 'dalton' 
CRS-2883: Resource 'ora.ctssd' failed during Clusterware stack start. 
CRS-4406: Oracle High Availability Services synchronous start failed. 
CRS-4000: Command Start failed, or completed with errors. 2017/02/18 10:21:41 
CLSRSC-117: Failed to start Oracle Clusterware stack Died at /u01/app/12.1.0/grid/crs/install/crsinstall.pm line 914.

The installation log is not much more useful: it just documents everything starting nicely until it fails for no discernible reason when trying to start ora.ctssd.

Take exactly the same two nodes and do the installation from the Dalton node, though, and everything just works -so it’s not, as I first thought it might be, something to do with networks, firewalls, DNS names resolution or the myriad other things that RAC depends on being ‘right’ before it will work. It’s purely and simply a matter of whether the local node’s name is longer or shorter than the remote node’s!

The problem is fixed in PSU 1 for, but it’s inappropriate to mandate its use in Churchill, since that’s supposed to work with the vanilla software available from OTN (I assume my readers lack support contracts, so everything has to work as-supplied from OTN for free).

The obvious fix for Churchill, therefore, is to (a) either make the ‘Gamow’ name one character longer (maybe spell it incorrectly as ‘gammow’?); or find a ‘D’ name that is both a physicist and only 4 characters long or fewer; or (c) change both names ensuring that the second is shorter than the first.

Largely due to the distinct lack of short-named, D-named physicists, I’ve gone for the (c) option: Churchill 1.7 therefore builds its Data Guard cluster using hosts geiger and dirac. Paul Dirac (that’s him on the top-left) was an English theoretical physicist, greatly admired by Richard Feynman (which makes him something of a star in these parts) and invented the relativistic equation of motion for the wave function of the electron. He used his equation to predict the existence of the positron -and of anti-matter in general, something for which he won a share of the 1933 Nobel prize for physics. Geiger is a frankly much less distinguished physicist whose main claim to fame is that he invented (most of) the Geiger counter and wasn’t (apparently) a Nazi. He gets into the Churchill Pantheon by the skin of his initial letter and not much else, to be honest!

Short version then: Churchill 1.7 now uses Alpher/Bethe and Geiger/Dirac clusters, and both Gamow and Dalton are no more. Quite a bit of documentation needs updating to take account of this trivial change! Hopefully, I should have that sorted by the end of the day. And that will teach me to test all parts of Churchill before declaring that ‘it works with 12c’. (Oooops!)

Recompile with -fPIC

Let me start by wishing a happy New Year to all my readers, complete with fireworks from our local council’s display!

And then let’s swiftly move on to the bad news!

If you are interested in installing Oracle onto non-approved Linux distros, you are very soon going to have to contend with this sort of error message:

/usr/bin/ld: /u01/app/oracle/product/12.1.0/db_1/lib//libpls12.a(pci.o): relocation R_X86_64_32 against `.rodata.str1.4' can not be used when making a shared object; recompile with -fPIC

This will be found in the Oracle installer’s log immediately after the “linking phase” of the Oracle installation starts.

Unfortunately, the error message dialog that appears at this point looks like this:

…and that particular error message has long been familiar from installs on assorted distros. The workarounds then were to add various compilation flags to assorted makefiles.

But in this case, the graphical error dialog deceives: for a start, this is happening on, and although the dialog text is the same as back in the days, the underlying cause is completely different. It’s only when you inspect the installActions log do you (eventually!) see the error text I showed above, which tells you that this is no “ordinary” compilation problem.

Welcome to the world of position-independent code.

Putting it as simply as I know how, the basic idea of position-independent code is that it allows execution of code regardless of its absolute memory address. It’s thus a ‘good thing’, on the whole.

Trouble is, if objects within the code you’re trying to compile haven’t themselves been compiled to be position-independent, then you aren’t yourself allowed to compile the code that references them into shared libraries.

As the error message above says, since “pci.o” isn’t position-independent, you can’t compile references to it into the libpls12 library. Note that the error message does not mean that your attempt to compile libpls12 should use fPIC: if it meant that, you could do something about it. No: it’s telling you that pci.o was compiled by Oracle without fPIC. Only if they re-compile that object with the fPIC compiler option switched on would you then be able to compile it into the libpls12 library successfully.

If you’re best mates with Larry, then, perhaps you’ll be able to get him to do the necessary recompilations for you! Mere mortals, however, are stuck with the unhappy fact that vast swathes of the Oracle 12c code-base has not been compiled to be position-independent… and there’s nothing you can do to fix that when your version of gcc insists that it should be.

The problem doesn’t manifest itself on all distros: Ubuntu 16.04, for example, has no problem installing 12c at all (apart from the usual ones associated with not using a supported distro, of course!) But Ubuntu 16.10 does have this fatal problem. Similarly, Debian 8 is fine with Oracle 12c, but Debian 9 (the testing branch, so not yet ready for mainstream release) fails. And whereas Manjaro didn’t have a problem earlier in the year when I first released my mercury pre-installer script, it does now.

This, of course, gives us a clue: there’s clearly some component of these distros which is being upgraded over time so that later releases fail where earlier ones didn’t. So what’s the upgraded component causing all the trouble?

Perhaps unsurprisingly, given that it’s a compilation error that shows you there’s a problem in the first place, it turns out that the gcc compiler is the culprit.

If you do a fresh install of Ubuntu 16.04 (the long-term support version, so still very much current and relevant), whilst making sure NOT to update anything as part of the installation process itself, issuing the command gcc -v will show you that version 5.4.0 is in use. Do the same thing on Ubuntu 16.10, however, and you’ll discover you’re now using gcc version 6.2.0.

A fresh Debian 8.6 install, subjected to an apt-get install gcc command,  ends up running gcc version 4.9.2. The same thing done to a fresh Debian 9 install results in a gcc version of 6.2.1.

Manjaro is a rolling release, of course, so it’s software components are forever being incrementally upgraded: it makes finding out what gcc version was in use at the start of the year rather tricky! So I don’t have hard evidence for the gcc version shift there -but my main desktop is currently reporting version 6.2.1, so I’ll stick my neck out and say that I would lay odds that, had I checked back in January 2016, I think I would have found it to be around version 5.something.

In short, for all three distros currently under my microscope, a shift from gcc 4- or 5-something to 6-something has taken place… and broken Oracle’s installation routine in the process.

It means that all distros will eventually come across this compilation problem as they eventually upgrade their gcc versions. Expect Fedora to keel over in short order, for example, when their version 26 is released next April (assuming they go for a late-release version of gcc 6.something, which I expect they will). No doubt we’ll have all moved on to installing Oracle 12c Release 2 by then, which is probably suitably position-independent throughout… so maybe no-one will ever have to worry about this issue again. But in the meantime… the constantly changing nature of gcc is a problem.

So, what’s to be done if you want Oracle 12c installed on these distros with their fancy new gcc versions? Nothing I could really think of, except to ensure that the old, functional versions of gcc and related development tools are installed …and that can be easier said than done!

On Debian 9 (‘testing’), for example, instead of just saying apt-get install gcc, you need to now say apt-get install gcc-5, which ensures the ‘right’ compiler version is installed, with which the Oracle installer can live. Thus can this screenshot be taken:

…which shows me happily querying the EMP table on whilst demonstrating that I’m running Debian Testing (codenamed “Stretch”). That’s only possible by careful curating of your development tool versions.

The same sort of ‘install old gcc version and make it the default’ trick is required to get 12c running on Ubuntu 16.10 too:

-though I had to specifically install gcc-4.9 rather than “gcc-5”, since the compilation error still arose when ‘gcc-5’ was installed. These things get tricky!

Anyway: there it is. Gcc’s constant version increments create havoc with Oracle 12c installations. Coming to a distro near you, soonish!

Fun Fedora 24

Just as my playing with the new Linux Mint release begins, so the Fedora team finalise a new version of their distro: Fedora 24 was released on 21st June.

It’s still very blue; it’s still very Gnome-y and therefore pretty awful as far as I’m concerned and I wouldn’t personally touch it with a feathered hat-band, let alone a bargepole.

But it’s out and therefore my Bogart preinstaller script, which makes Fedora a suitable platform for running Oracle Enterprise Edition, needs a run in the park to make sure it still works with the new version. Happily it does without any substantial changes at all.

However, I took the opportunity to do two things with Bogart. One was to remove its ability for preparing for an 11g installation. I know is still supported, but you can’t get hold of that without a support contract; and if you’ve got a support contract, you won’t likely be wanting to run Oracle on an unsupported platform like Fedora! Meanwhile, any other version of 11g you can get your hands on has long-since been de-suppported… so Bogart is now 12c only.

And that means, two: I’ve re-written the Oracle-on-Fedora article to reflect it’s new only-12c-ness.

The revised article is here, and the updated Bogart preinstaller script is here.

Churchill on Windows?!

I’ve had many requests over the years to repeat my ‘Churchill Framework’ on Windows, “Churchill” being my mostly-automated way of building a virtual RAC using Linux as the operating system of choice.

I’ve always refused: if you want a desktop RAC on your Windows PC, why not just deploy Churchill ‘proper’ and have three virtual machines running CentOS. It’s a RAC, and it’s still “on” Windows, isn’t it?!

Well, of course, that wasn’t quite the point my correspondents were making. They wanted a desktop RAC running on top of purely Windows operating systems. They aren’t Linux users, and they’re not interested in working at a command line. Could I please oblige?

Again, I’ve always said no, because Windows costs lots of money. It’s easy to build a 3-node or a 6-node setup in Linux, because you aren’t paying $1000 a pop every time you install your operating system! It seemed to me that RAC-on-Windows was a nice idea (I had it working back in 2001 with 9i on Windows 2000 after all), but it wasn’t very practical as a learning platform.

Happily for my correspondents, I’ve now changed my view in that regard. All the Windows-based would-be DBAs of my acquaintance are working for companies that supply them with MSDN subscriptions. And Microsoft’s Technet evaluation options allow even people with no MSDN access to download and use Windows Server 2012 and beyond for free, for at least 6 months.

So I’ve given in. There’s now available a new article for doing Desktop RAC using nothing but Windows. It bears a passing resemblance to ‘proper’ Churchill: there are three servers to build, with one acting as the supplier of shared storage and needed network services to the others. There’s even the use of iSCSI to provide the virtual shared storage layer. But it’s about as non-Churchill as it gets, really, because everything is hand-built… which explains the enormous number of screenshots and the overall length of the article!

Oracle on Linux Mint 17.2

I am casting my eyes around for a new desktop operating system (yet again!). I’m currently running Windows 10, and whilst it’s OK, there are some privacy issues I have concerns about. So, though I’m happy-ish with the current status quo, I am doing some experiments on the side for an alternative.

Linux Mint is right up there with the best of them as a suitable desktop OS. In the past, I’ve run Linux Mint Debian Edition (LMDE), but this time I thought I’d take the ‘standard’ version for a run: it’s based on Ubuntu.

Of course, getting Oracle 12c running on any form of non-Red Hat/non-CentOS distro can be tricky, and although my old Gladstone script used to automate it for a number of ‘peripheral’ distros, I haven’t maintained it for a while.

So I’ve written a new automation script, stripping out a lot of the complexity from the original gladstone script in the process. The new script configures a fresh 64-bit Linux Mint 17.2 desktop for an Oracle 64-bit database installation, making the user that runs it the “oracle user” and owner of the resulting Oracle software installation. By running it, a “linking-error-fix.sh” script is written to the user’s desktop, for later use at the point when the Oracle software installation starts to throw up linking errors.

Installing Oracle on LM17.2 thus becomes a matter of downloading and unpacking the software from Oracle and the oracleonmint.sh script from me, double-clicking the shell script and supplying the root password and waiting for a lot of prerequisite software to be downloaded from the standard Linux Mint repositories. When that’s all finished, you’ll be prompted to reboot.

Once your PC is back up and running, you just launch Oracle’s own runInstaller as usual, click [Next] lots of times, and wait for the inevitable linking errors to occur once the installer tries to build Oracle binaries.

At that point, you just double-click the linking-error-fix.sh script that should be sitting on your desktop, click [Retry] in the Oracle installer …and wait for the installation and database creation process to complete. Update: There’s now a full-blown article documenting what you do, step by step.

No messing around with editing various obscure compiler files by hand: just run the first shell script to create the second; and run the second when the Oracle installer throws up its first linking error.

I’m running out of prime ministers these days, so this new script doesn’t have a fancy name. It’s just my “Oracle on Mint” script :-) It has been tested on all three of the Cinnamon, Mate and KDE versions of Linux Mint 17.2. There are no differences in how it runs on any of them.

Whether or not I do end up ditching Windows 10 for LM17.2, I can’t yet say: but being able to run Oracle on LM17.2 certainly makes the idea of a transition a whole lot more feasible.