Atlas and 12cR2

Continuing the saga of “catching up” with software developments whilst I was en route to the UK from Australia, the question next to be addressed is simply: does Atlas work to allow Oracle 12c Release 2 to be installed onto the many and various distros it claims to support.

The answer is a bit mixed, I’m afraid!

First thing to say: I deliberately coded Atlas to specifically declare 12cR2 ‘didn’t work yet’ when I first released it. So, for it to work for 12cR2 at all, that bit of code has to be removed -and that means Atlas straightaway gets bumped to version 1.2.

Once that code has gone, the good news is that Atlas will successfully help 12cR2 be installed on the majority of the distros it works with (15 out of 19, as it happens). But the bad news is that if your distro is Ubuntu or even just Ubuntu-based, then the linking phase of the Oracle 12c Release 2 installation fails irreparably after Atlas has been run.

The complete list of distros and their results with Atlas for 12c Release 2 is therefore as follows:

Debian 8.2+ ............................ Works fine
Linux Mint 18+ ......................... Fails
Mint Debian Edition 2+ ................. Works fine
Red Hat ES 7.0+ ........................ Works fine
Scientific Linux 7.0+ .................. Works fine
CentOS 7.0+ ............................ Works fine
OpenSuse Leap 42+ ...................... Works fine
Antergos  2016.11+ ..................... Works fine
elementary OS 0.4+ ..................... Fails
Mageia 5+ .............................. Works fine
Korora 25+ ............................. Works fine
Zorin Core 12 .......................... Fails
Ubuntu 16+ ............................. Fails
Manjaro 15+ ............................ Works fine 
Fedora 23+ ............................. Works fine
Peppermint Linux 7+ .................... Fails
GeckoLinux Static 422+ ................. Works fine
Chapeau Linux 24+ ...................... Works fine
PCLinuxOS 2016+ ........................ Works fine

As I say, the pattern is pretty obvious and I suspect the problem is that the gcc versioning tricks I had to pull to get 12cR1 to compile properly on any Ubuntu-based distro back in January are now the cause of woes for 12cR2. The failure always manifests itself in the following manner:

There’s no recovering from that as yet, but I hope to get it sorted within the week.

On the other hand, for any distro listed as working fine in the above list, you can expect the usual plain sailing:

That’s specifically Oracle 12.2.0.1 running on GeckoLinux, which is a spin of openSUSE Tumbleweed, but you get the same outcome for all non-Ubuntu distros.

Whilst testing all this, I discovered a couple of distros which had incremented their version strings since January (Manjaro, for instance, now reports itself as version ’17.something’, so the part of Atlas where it checked for version strings containing the numbers 15 or 16 obviously needed updating). Those sorts of versioning updates are now also included in Atlas 1.2.

There is very little updating of the Atlas doco to do, happily. For a start, you will obtain the new Atlas version just by running exactly the same wget command as was previously documented: the URL alias simply points to the latest version, but the URL itself doesn’t change.

When you install Oracle 12cR2 onto any of these non-standard distros (except the RHCSL ones, of course), you will get this dire-looking warning:

It’s better than the 12cR1 equivalent, which was to say ‘your system is inadequate’! Anyway, for all the distros for which Atlas works at all, it’s perfectly OK to say ‘yes, I want to continue’. The installation will succeed anyway.

Have fun… and wish me luck whilst battling with Ubuntu later this week 🙂

Atlas, Ubuntu 17 and 12cR1

As I mentioned last post, one major issue that has arisen since I left Australia is this: does Atlas run on the freshly-released Ubuntu 17.04 to make Oracle 12c Release 1 (12.1.0.2) installations a piece of cake? It was working fine on 16.04 and 16.10; but would things break with the fresh 17.04 release?

Here’s the answer:

To be fair, I had to alter the Atlas script in one tiny respect: it originally tested for the existence of ’16’ as one component of the distro’s version string. That was obviously a bit restrictive! I’ve now added ’17’ as a test, too… and that means Atlas’s version bumps to 1.1.

You don’t need to do anything different than was originally documented, though: the ‘wget http://bit.do/dizatlas -O atlas.sh‘ instruction works as it always did. It’s just that it now downloads the newer Atlas script instead of the original.

Timing…

If the secret of comedy is timing, then Oracle must be the funniest corporation around!

With less than 24 hours to go before my flight back to the UK, they make 12c Release 2 available for general use.

Naturally, all my PCs and servers are packed, so I only have a feeble airplane-ready spare to do anything on. I have managed a 12cR2 install using Atlas on CentOS 7, so I can at least confirm the new version doesn’t really change anything much as far as the installation process is concerned and hardly anything about Atlas needs to be modified to deal with it. But I haven’t tested all the other distros Atlas claims to work with, so I’m not releasing a 12cR2-ready version of Atlas yet. First order of business when I reach Blighty’s shores, I guess.

The major drama as far as 12cR2 is concerned, from my point of view, is that the EMP table doesn’t exist by default any more 🙁 (but rdbms/admin/utlsampl.sql will create it if needed, as it has always done). A sad day indeed, then.

Plus there are thunderstorms forecast for Sydney tomorrow. It’ll be a bumpy take-off…

Finished

Finished! And only three weeks late!

I’m referring to Churchill 1.7, a major overhaul of the Churchill framework and its accompanying documentation, making it work on RHCSL 6.8, for 12c only, with Enterprise Manager Cloud Control 13c and a complete overhaul of the bootstrap options available when kickstarting O/S installations.

At some point toward the end of January, it morphed into practically a complete re-write… and I thought seriously about calling it quits and declaring it to be version 2.0. But I’ve stuck with the incremental versioning for now. (I’ve been saving version 2 for when I get round to making it work with RHCSL 7.x distros).

I’m finished in another sense, though, too: the contract to purchase a house in Nottingham is ready to sign and it accordingly looks very much as though I’ll be becoming an ex-Aussie (or a re-Englishman, I suppose, depending on your point of view) on or around 6th March. I may not have much time to post much here given the packing, flight-booking, passport-checking, Internet banking, etc etc shenanigans that now ensue. If I can I will, but otherwise I’ll be back online toward the end of March, live from Nottingham 🙂

What were the odds?

In modernising Churchill to work for Oracle 12c and the latest 6.x releases of RHCSL, I’ve encountered a bizarre bug (#19476913 if you’re able to check up on it), whereby startup of the cluster stack on a remote node fails if its hostname is longer than (or equal to) the hostname of the local node.

That is, if you are running the Grid Infrastructure installer from Alpher (6 characters) and pushing to Bethe (5 characters) then the CRS starts on Bethe just fine: local 6 is greater than remote 5. But if you are running the GI installer on Gamow (5 characters) and pushing to Dalton (6 characters) then the installer’s attempt to restart the CRS on Dalton will fail, since now local 5 is less than remote 6. Alpher/Bethe managed to dodge this bullet, of course -but only by pure luck.

The symptoms are that during the installation of Grid Infrastructure, all works well until the root scripts are run, at which point (and after a long wait), this pops up:

Poke around in the [Details] of that dialog and you’ll see this:

CRS-2676: Start of 'ora.cssdmonitor' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cssd' on 'dalton' 
CRS-2672: Attempting to start 'ora.diskmon' on 'dalton' 
CRS-2676: Start of 'ora.diskmon' on 'dalton' succeeded 
CRS-2676: Start of 'ora.cssd' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dalton' 
CRS-2672: Attempting to start 'ora.ctssd' on 'dalton' 
CRS-2883: Resource 'ora.ctssd' failed during Clusterware stack start. 
CRS-4406: Oracle High Availability Services synchronous start failed. 
CRS-4000: Command Start failed, or completed with errors. 2017/02/18 10:21:41 
CLSRSC-117: Failed to start Oracle Clusterware stack Died at /u01/app/12.1.0/grid/crs/install/crsinstall.pm line 914.

The installation log is not much more useful: it just documents everything starting nicely until it fails for no discernible reason when trying to start ora.ctssd.

Take exactly the same two nodes and do the installation from the Dalton node, though, and everything just works -so it’s not, as I first thought it might be, something to do with networks, firewalls, DNS names resolution or the myriad other things that RAC depends on being ‘right’ before it will work. It’s purely and simply a matter of whether the local node’s name is longer or shorter than the remote node’s!

The problem is fixed in PSU 1 for 12.1.0.2, but it’s inappropriate to mandate its use in Churchill, since that’s supposed to work with the vanilla software available from OTN (I assume my readers lack support contracts, so everything has to work as-supplied from OTN for free).

The obvious fix for Churchill, therefore, is to (a) either make the ‘Gamow’ name one character longer (maybe spell it incorrectly as ‘gammow’?); or find a ‘D’ name that is both a physicist and only 4 characters long or fewer; or (c) change both names ensuring that the second is shorter than the first.

Largely due to the distinct lack of short-named, D-named physicists, I’ve gone for the (c) option: Churchill 1.7 therefore builds its Data Guard cluster using hosts geiger and dirac. Paul Dirac (that’s him on the top-left) was an English theoretical physicist, greatly admired by Richard Feynman (which makes him something of a star in these parts) and invented the relativistic equation of motion for the wave function of the electron. He used his equation to predict the existence of the positron -and of anti-matter in general, something for which he won a share of the 1933 Nobel prize for physics. Geiger is a frankly much less distinguished physicist whose main claim to fame is that he invented (most of) the Geiger counter and wasn’t (apparently) a Nazi. He gets into the Churchill Pantheon by the skin of his initial letter and not much else, to be honest!

Short version then: Churchill 1.7 now uses Alpher/Bethe and Geiger/Dirac clusters, and both Gamow and Dalton are no more. Quite a bit of documentation needs updating to take account of this trivial change! Hopefully, I should have that sorted by the end of the day. And that will teach me to test all parts of Churchill before declaring that ‘it works with 12c’. (Oooops!)

Churchill Changes

I’ve just published version 1.7 of Churchill. It contains a lot of changes.

The main one is that I’ve removed a few dependencies on .i686 packages. That means the O/S installations can now all take place off the first DVD alone. No second DVD is prompted for, in other words.

This in turn brings about the biggest single benefit of the new release: it works with CentOS 6.8 (and Scientific Linux 6.8, too).

In fact, Churchill 1.7 now reverts to making CentOS 6.8 the default O/S assumed to be in use. (You can still always specify ‘redhat’, ‘sl’ or ‘oel’ if you prefer to use real Red Hat, Scientific Linux or Oracle Enterprise Linux, of course; and you can always specify an earlier distro version if you prefer to stick with (say) 6.3 -though I can’t think why you’d particularly want to).

The other big change is that the bootstrap lines are now trivially easy. Back in 2013 when I first released Churchill, it seemed like a good idea to make it as flexible as possible so that users could specify their own IP addresses and hostnames; but this just made for really lengthy bootstrap lines and confused the heck out of everybody!

So, the simplify brush has been daubed all over Churchill. You now must use the speed keys 1 to 4 as you build your nodes (or you can instead specify their corresponding hostnames). By specifying the speed key or hostname, you automatically define all the pesky details about IP address and interconnect IP address. It makes things a lot simpler and less confusing, I think. It also makes it a bit less flexible… but that’s the price you pay for simplicity. Ask the Gnome developers!

Other changes flow in consequence: the filecopy=y/n parameter is no longer required. If you are building nodes 1 or 3, file copying is assumed to be ‘yes’; if you are building nodes 2 or 4, it’s assumed to be ‘no’. Likewise, there’s no dg=y/n parameter any more: if you are building nodes 1 or 2, it’s assumed to be ‘no’; but build nodes 3 or 4, it’s assumed to be ‘yes’.

It is still possible to say sk=1&rac=n (or hostname=alpher&rac=n), though: that’s if you are building node 1 but want to use it in standalone mode.

As with the previous release (1.6), Churchill only works to build 12c standalone and RAC/Data Guard environments: there is no Oracle 11g support these days.

These are substantial changes and mean Churchill now works in ways quite different from before. That obviously affects the way it is documented. Those documentation changes are being made as I write and should be ‘live’ by the time you read this. Since the old versions of Churchill are kept available on the old site, I’ll keep the original documentation available on that old site too, at least for now.

On this site, however, only the 1.7 version will be available and documented.

Murdock

I can’t say I’ve ever used Debian for long, but it has always proved itself a worthy contender in any distro comparison I’ve ever run -largely because it’s stable, has an enormous number of packages available to it and has its own quasi-minimalist graphical charm. If it lacks some of the bells and whistles of the more glamorous distros (many of which, such as Ubuntu -and from there Mint- are actually derived from Debian), it is nevertheless a strong contender.

How lucky we are that Ian Murdock saw fit to create it (and then go on to champion open source software in various other ways after that), back in 1993. And what a shock it was to read on New Year’s morning that he’d been found dead on 28th December.

The terrible news prompted me to look at my old Gladstone script for installing Oracle on various non-standard Linux distros and discovered that the last version of Debian I had it working for was 6.0.6, which dates it to around September 2012. I thought the least I could do is to bring it a bit more up-to-date than that.

So I’m releasing a new pre-installer script, called murdock, extricated from the ancient ruins of Gladstone and made functional once more. It prepares a Debian 8.2 64-bit system that has a functioning Internet connection for running either Oracle 11g or 12c. It doesn’t work for any other distro, nor any other bit-ness. (An alternative download location is here).

I’ll knock together some better instructions at some point soon, but basically download it, run it, supply the root password and sit back to watch everything get configured appropriately. Then download the Oracle software and run it. It will fail nearly all its prerequisites checks, of course. Ignore them all. It will then error during the linking phase: but murdock has anticipated that and created a fix-up script in the oracle user’s Documents directory: so run that script, click [Retry] in the Oracle installer, and everything will work correctly thereafter.

(Update: the ‘better instructions’ are now an article available here.)

I’ve tested it on both 11.2.0.1, 11.2.0.3, 11.2.0.4 and 12.1.0.2, using the Gnome, Cinnamon, MATE and KDE desktop environments. Gnome has some dialog problems that make the installer non-functional unless invoked remotely (i.e., use ssh -X to make a remote connection to the server and run the installer from the remote PC). And 11.2.0.1 has a compilation error that can’t be easily fixed, meaning that Oracle Text indexes can’t work. Otherwise, all works as expected.

I don’t run single-node Oracle much these days. I don’t run anything that’s not on Windows or Solaris come to that. But Open Source in general, and Debian specifically, has given me so much, it seemed to be at least something I could do by way of paying it back a little. Any issues with it, let me know…

Churchill on Windows?!

I’ve had many requests over the years to repeat my ‘Churchill Framework’ on Windows, “Churchill” being my mostly-automated way of building a virtual RAC using Linux as the operating system of choice.

I’ve always refused: if you want a desktop RAC on your Windows PC, why not just deploy Churchill ‘proper’ and have three virtual machines running CentOS. It’s a RAC, and it’s still “on” Windows, isn’t it?!

Well, of course, that wasn’t quite the point my correspondents were making. They wanted a desktop RAC running on top of purely Windows operating systems. They aren’t Linux users, and they’re not interested in working at a command line. Could I please oblige?

Again, I’ve always said no, because Windows costs lots of money. It’s easy to build a 3-node or a 6-node setup in Linux, because you aren’t paying $1000 a pop every time you install your operating system! It seemed to me that RAC-on-Windows was a nice idea (I had it working back in 2001 with 9i on Windows 2000 after all), but it wasn’t very practical as a learning platform.

Happily for my correspondents, I’ve now changed my view in that regard. All the Windows-based would-be DBAs of my acquaintance are working for companies that supply them with MSDN subscriptions. And Microsoft’s Technet evaluation options allow even people with no MSDN access to download and use Windows Server 2012 and beyond for free, for at least 6 months.

So I’ve given in. There’s now available a new article for doing Desktop RAC using nothing but Windows. It bears a passing resemblance to ‘proper’ Churchill: there are three servers to build, with one acting as the supplier of shared storage and needed network services to the others. There’s even the use of iSCSI to provide the virtual shared storage layer. But it’s about as non-Churchill as it gets, really, because everything is hand-built… which explains the enormous number of screenshots and the overall length of the article!

Oracle 12.1.0.2 on Linux Mint 17.2

I am casting my eyes around for a new desktop operating system (yet again!). I’m currently running Windows 10, and whilst it’s OK, there are some privacy issues I have concerns about. So, though I’m happy-ish with the current status quo, I am doing some experiments on the side for an alternative.

Linux Mint is right up there with the best of them as a suitable desktop OS. In the past, I’ve run Linux Mint Debian Edition (LMDE), but this time I thought I’d take the ‘standard’ version for a run: it’s based on Ubuntu.

Of course, getting Oracle 12c running on any form of non-Red Hat/non-CentOS distro can be tricky, and although my old Gladstone script used to automate it for a number of ‘peripheral’ distros, I haven’t maintained it for a while.

So I’ve written a new automation script, stripping out a lot of the complexity from the original gladstone script in the process. The new script configures a fresh 64-bit Linux Mint 17.2 desktop for an Oracle 12.1.0.2 64-bit database installation, making the user that runs it the “oracle user” and owner of the resulting Oracle software installation. By running it, a “linking-error-fix.sh” script is written to the user’s desktop, for later use at the point when the Oracle software installation starts to throw up linking errors.

Installing Oracle on LM17.2 thus becomes a matter of downloading and unpacking the software from Oracle and the oracleonmint.sh script from me, double-clicking the shell script and supplying the root password and waiting for a lot of prerequisite software to be downloaded from the standard Linux Mint repositories. When that’s all finished, you’ll be prompted to reboot.

Once your PC is back up and running, you just launch Oracle’s own runInstaller as usual, click [Next] lots of times, and wait for the inevitable linking errors to occur once the installer tries to build Oracle binaries.

At that point, you just double-click the linking-error-fix.sh script that should be sitting on your desktop, click [Retry] in the Oracle installer …and wait for the installation and database creation process to complete. Update: There’s now a full-blown article documenting what you do, step by step.

No messing around with editing various obscure compiler files by hand: just run the first shell script to create the second; and run the second when the Oracle installer throws up its first linking error.

I’m running out of prime ministers these days, so this new script doesn’t have a fancy name. It’s just my “Oracle on Mint” script :-) It has been tested on all three of the Cinnamon, Mate and KDE versions of Linux Mint 17.2. There are no differences in how it runs on any of them.

Whether or not I do end up ditching Windows 10 for LM17.2, I can’t yet say: but being able to run Oracle on LM17.2 certainly makes the idea of a transition a whole lot more feasible.

When I’m done…

I hesitate to draw comparisons between me and Michaelangelo doing the Sistine Chapel, but I am reminded of Rex Harrison (ok, the Pope) forever asking Charlton Heston (ok, Michaelangelo): “When will you make an end of it!” And Heston,Michaelangelo tartly replying, “When I’m done!”

Well, we’re finished: the Wilson server article has just been completed, and only a week overdue :-)

And that completes the Churchill framework. Use it to its full and you now end up with two two-node clusters, one of which becomes a primary 2-node RAC, leaving the other to become a 2-node RAC running a standby copy of the primary; one network services server and its backup (Churchill and Attlee), and a beefy Enterprise Manager server to keep an eye on everything else (Wilson).

It’s quite a nice environment in which to try out things like patching, failover and switchover, configuring Cloud Control to send meaningful alerts, and so on. Happily, I’ve used 11.2.0.3, 11.2.0.4 and 12.1.0.2 with equally good results, and on top of CentOS 6.5 and 6.6, with an OEL 6.5 in there somewhere too.

My main desktop runs this complete infrastructure very nicely; my Toshiba (16GB RAM, 1.5TB solid state HDD) runs it just as nicely, though I have to dial down the memory numbers for my database VMs. My poor, ageing HP Folio 13 (8GB RAM, 256GB Solid State) has no problem with a 2-node RAC and an Enterprise Manager, but gettng it to do an additional 2 nodes for Active Data Guard practice is pushing it a bit. Maybe I should buy more solid state hard drive for it?! There’s a thought…

It’s everything I’ve ever wanted in a ‘build Oracle properly, routinely, accurately, automatically’ tool.

So now I can stop…