Timing…

If the secret of comedy is timing, then Oracle must be the funniest corporation around!

With less than 24 hours to go before my flight back to the UK, they make 12c Release 2 available for general use.

Naturally, all my PCs and servers are packed, so I only have a feeble airplane-ready spare to do anything on. I have managed a 12cR2 install using Atlas on CentOS 7, so I can at least confirm the new version doesn’t really change anything much as far as the installation process is concerned and hardly anything about Atlas needs to be modified to deal with it. But I haven’t tested all the other distros Atlas claims to work with, so I’m not releasing a 12cR2-ready version of Atlas yet. First order of business when I reach Blighty’s shores, I guess.

The major drama as far as 12cR2 is concerned, from my point of view, is that the EMP table doesn’t exist by default any more 🙁 (but rdbms/admin/utlsampl.sql will create it if needed, as it has always done). A sad day indeed, then.

Plus there are thunderstorms forecast for Sydney tomorrow. It’ll be a bumpy take-off…

Finished

Finished! And only three weeks late!

I’m referring to Churchill 1.7, a major overhaul of the Churchill framework and its accompanying documentation, making it work on RHCSL 6.8, for 12c only, with Enterprise Manager Cloud Control 13c and a complete overhaul of the bootstrap options available when kickstarting O/S installations.

At some point toward the end of January, it morphed into practically a complete re-write… and I thought seriously about calling it quits and declaring it to be version 2.0. But I’ve stuck with the incremental versioning for now. (I’ve been saving version 2 for when I get round to making it work with RHCSL 7.x distros).

I’m finished in another sense, though, too: the contract to purchase a house in Nottingham is ready to sign and it accordingly looks very much as though I’ll be becoming an ex-Aussie (or a re-Englishman, I suppose, depending on your point of view) on or around 6th March. I may not have much time to post much here given the packing, flight-booking, passport-checking, Internet banking, etc etc shenanigans that now ensue. If I can I will, but otherwise I’ll be back online toward the end of March, live from Nottingham 🙂

What were the odds?

In modernising Churchill to work for Oracle 12c and the latest 6.x releases of RHCSL, I’ve encountered a bizarre bug (#19476913 if you’re able to check up on it), whereby startup of the cluster stack on a remote node fails if its hostname is longer than (or equal to) the hostname of the local node.

That is, if you are running the Grid Infrastructure installer from Alpher (6 characters) and pushing to Bethe (5 characters) then the CRS starts on Bethe just fine: local 6 is greater than remote 5. But if you are running the GI installer on Gamow (5 characters) and pushing to Dalton (6 characters) then the installer’s attempt to restart the CRS on Dalton will fail, since now local 5 is less than remote 6. Alpher/Bethe managed to dodge this bullet, of course -but only by pure luck.

The symptoms are that during the installation of Grid Infrastructure, all works well until the root scripts are run, at which point (and after a long wait), this pops up:

Poke around in the [Details] of that dialog and you’ll see this:

CRS-2676: Start of 'ora.cssdmonitor' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cssd' on 'dalton' 
CRS-2672: Attempting to start 'ora.diskmon' on 'dalton' 
CRS-2676: Start of 'ora.diskmon' on 'dalton' succeeded 
CRS-2676: Start of 'ora.cssd' on 'dalton' succeeded 
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'dalton' 
CRS-2672: Attempting to start 'ora.ctssd' on 'dalton' 
CRS-2883: Resource 'ora.ctssd' failed during Clusterware stack start. 
CRS-4406: Oracle High Availability Services synchronous start failed. 
CRS-4000: Command Start failed, or completed with errors. 2017/02/18 10:21:41 
CLSRSC-117: Failed to start Oracle Clusterware stack Died at /u01/app/12.1.0/grid/crs/install/crsinstall.pm line 914.

The installation log is not much more useful: it just documents everything starting nicely until it fails for no discernible reason when trying to start ora.ctssd.

Take exactly the same two nodes and do the installation from the Dalton node, though, and everything just works -so it’s not, as I first thought it might be, something to do with networks, firewalls, DNS names resolution or the myriad other things that RAC depends on being ‘right’ before it will work. It’s purely and simply a matter of whether the local node’s name is longer or shorter than the remote node’s!

The problem is fixed in PSU 1 for 12.1.0.2, but it’s inappropriate to mandate its use in Churchill, since that’s supposed to work with the vanilla software available from OTN (I assume my readers lack support contracts, so everything has to work as-supplied from OTN for free).

The obvious fix for Churchill, therefore, is to (a) either make the ‘Gamow’ name one character longer (maybe spell it incorrectly as ‘gammow’?); or find a ‘D’ name that is both a physicist and only 4 characters long or fewer; or (c) change both names ensuring that the second is shorter than the first.

Largely due to the distinct lack of short-named, D-named physicists, I’ve gone for the (c) option: Churchill 1.7 therefore builds its Data Guard cluster using hosts geiger and dirac. Paul Dirac (that’s him on the top-left) was an English theoretical physicist, greatly admired by Richard Feynman (which makes him something of a star in these parts) and invented the relativistic equation of motion for the wave function of the electron. He used his equation to predict the existence of the positron -and of anti-matter in general, something for which he won a share of the 1933 Nobel prize for physics. Geiger is a frankly much less distinguished physicist whose main claim to fame is that he invented (most of) the Geiger counter and wasn’t (apparently) a Nazi. He gets into the Churchill Pantheon by the skin of his initial letter and not much else, to be honest!

Short version then: Churchill 1.7 now uses Alpher/Bethe and Geiger/Dirac clusters, and both Gamow and Dalton are no more. Quite a bit of documentation needs updating to take account of this trivial change! Hopefully, I should have that sorted by the end of the day. And that will teach me to test all parts of Churchill before declaring that ‘it works with 12c’. (Oooops!)

Churchill Changes

I’ve just published version 1.7 of Churchill. It contains a lot of changes.

The main one is that I’ve removed a few dependencies on .i686 packages. That means the O/S installations can now all take place off the first DVD alone. No second DVD is prompted for, in other words.

This in turn brings about the biggest single benefit of the new release: it works with CentOS 6.8 (and Scientific Linux 6.8, too).

In fact, Churchill 1.7 now reverts to making CentOS 6.8 the default O/S assumed to be in use. (You can still always specify ‘redhat’, ‘sl’ or ‘oel’ if you prefer to use real Red Hat, Scientific Linux or Oracle Enterprise Linux, of course; and you can always specify an earlier distro version if you prefer to stick with (say) 6.3 -though I can’t think why you’d particularly want to).

The other big change is that the bootstrap lines are now trivially easy. Back in 2013 when I first released Churchill, it seemed like a good idea to make it as flexible as possible so that users could specify their own IP addresses and hostnames; but this just made for really lengthy bootstrap lines and confused the heck out of everybody!

So, the simplify brush has been daubed all over Churchill. You now must use the speed keys 1 to 4 as you build your nodes (or you can instead specify their corresponding hostnames). By specifying the speed key or hostname, you automatically define all the pesky details about IP address and interconnect IP address. It makes things a lot simpler and less confusing, I think. It also makes it a bit less flexible… but that’s the price you pay for simplicity. Ask the Gnome developers!

Other changes flow in consequence: the filecopy=y/n parameter is no longer required. If you are building nodes 1 or 3, file copying is assumed to be ‘yes’; if you are building nodes 2 or 4, it’s assumed to be ‘no’. Likewise, there’s no dg=y/n parameter any more: if you are building nodes 1 or 2, it’s assumed to be ‘no’; but build nodes 3 or 4, it’s assumed to be ‘yes’.

It is still possible to say sk=1&rac=n (or hostname=alpher&rac=n), though: that’s if you are building node 1 but want to use it in standalone mode.

As with the previous release (1.6), Churchill only works to build 12c standalone and RAC/Data Guard environments: there is no Oracle 11g support these days.

These are substantial changes and mean Churchill now works in ways quite different from before. That obviously affects the way it is documented. Those documentation changes are being made as I write and should be ‘live’ by the time you read this. Since the old versions of Churchill are kept available on the old site, I’ll keep the original documentation available on that old site too, at least for now.

On this site, however, only the 1.7 version will be available and documented.

Murdock

I can’t say I’ve ever used Debian for long, but it has always proved itself a worthy contender in any distro comparison I’ve ever run -largely because it’s stable, has an enormous number of packages available to it and has its own quasi-minimalist graphical charm. If it lacks some of the bells and whistles of the more glamorous distros (many of which, such as Ubuntu -and from there Mint- are actually derived from Debian), it is nevertheless a strong contender.

How lucky we are that Ian Murdock saw fit to create it (and then go on to champion open source software in various other ways after that), back in 1993. And what a shock it was to read on New Year’s morning that he’d been found dead on 28th December.

The terrible news prompted me to look at my old Gladstone script for installing Oracle on various non-standard Linux distros and discovered that the last version of Debian I had it working for was 6.0.6, which dates it to around September 2012. I thought the least I could do is to bring it a bit more up-to-date than that.

So I’m releasing a new pre-installer script, called murdock, extricated from the ancient ruins of Gladstone and made functional once more. It prepares a Debian 8.2 64-bit system that has a functioning Internet connection for running either Oracle 11g or 12c. It doesn’t work for any other distro, nor any other bit-ness. (An alternative download location is here).

I’ll knock together some better instructions at some point soon, but basically download it, run it, supply the root password and sit back to watch everything get configured appropriately. Then download the Oracle software and run it. It will fail nearly all its prerequisites checks, of course. Ignore them all. It will then error during the linking phase: but murdock has anticipated that and created a fix-up script in the oracle user’s Documents directory: so run that script, click [Retry] in the Oracle installer, and everything will work correctly thereafter.

(Update: the ‘better instructions’ are now an article available here.)

I’ve tested it on both 11.2.0.1, 11.2.0.3, 11.2.0.4 and 12.1.0.2, using the Gnome, Cinnamon, MATE and KDE desktop environments. Gnome has some dialog problems that make the installer non-functional unless invoked remotely (i.e., use ssh -X to make a remote connection to the server and run the installer from the remote PC). And 11.2.0.1 has a compilation error that can’t be easily fixed, meaning that Oracle Text indexes can’t work. Otherwise, all works as expected.

I don’t run single-node Oracle much these days. I don’t run anything that’s not on Windows or Solaris come to that. But Open Source in general, and Debian specifically, has given me so much, it seemed to be at least something I could do by way of paying it back a little. Any issues with it, let me know…

Churchill on Windows?!

I’ve had many requests over the years to repeat my ‘Churchill Framework’ on Windows, “Churchill” being my mostly-automated way of building a virtual RAC using Linux as the operating system of choice.

I’ve always refused: if you want a desktop RAC on your Windows PC, why not just deploy Churchill ‘proper’ and have three virtual machines running CentOS. It’s a RAC, and it’s still “on” Windows, isn’t it?!

Well, of course, that wasn’t quite the point my correspondents were making. They wanted a desktop RAC running on top of purely Windows operating systems. They aren’t Linux users, and they’re not interested in working at a command line. Could I please oblige?

Again, I’ve always said no, because Windows costs lots of money. It’s easy to build a 3-node or a 6-node setup in Linux, because you aren’t paying $1000 a pop every time you install your operating system! It seemed to me that RAC-on-Windows was a nice idea (I had it working back in 2001 with 9i on Windows 2000 after all), but it wasn’t very practical as a learning platform.

Happily for my correspondents, I’ve now changed my view in that regard. All the Windows-based would-be DBAs of my acquaintance are working for companies that supply them with MSDN subscriptions. And Microsoft’s Technet evaluation options allow even people with no MSDN access to download and use Windows Server 2012 and beyond for free, for at least 6 months.

So I’ve given in. There’s now available a new article for doing Desktop RAC using nothing but Windows. It bears a passing resemblance to ‘proper’ Churchill: there are three servers to build, with one acting as the supplier of shared storage and needed network services to the others. There’s even the use of iSCSI to provide the virtual shared storage layer. But it’s about as non-Churchill as it gets, really, because everything is hand-built… which explains the enormous number of screenshots and the overall length of the article!

Oracle 12.1.0.2 on Linux Mint 17.2

I am casting my eyes around for a new desktop operating system (yet again!). I’m currently running Windows 10, and whilst it’s OK, there are some privacy issues I have concerns about. So, though I’m happy-ish with the current status quo, I am doing some experiments on the side for an alternative.

Linux Mint is right up there with the best of them as a suitable desktop OS. In the past, I’ve run Linux Mint Debian Edition (LMDE), but this time I thought I’d take the ‘standard’ version for a run: it’s based on Ubuntu.

Of course, getting Oracle 12c running on any form of non-Red Hat/non-CentOS distro can be tricky, and although my old Gladstone script used to automate it for a number of ‘peripheral’ distros, I haven’t maintained it for a while.

So I’ve written a new automation script, stripping out a lot of the complexity from the original gladstone script in the process. The new script configures a fresh 64-bit Linux Mint 17.2 desktop for an Oracle 12.1.0.2 64-bit database installation, making the user that runs it the “oracle user” and owner of the resulting Oracle software installation. By running it, a “linking-error-fix.sh” script is written to the user’s desktop, for later use at the point when the Oracle software installation starts to throw up linking errors.

Installing Oracle on LM17.2 thus becomes a matter of downloading and unpacking the software from Oracle and the oracleonmint.sh script from me, double-clicking the shell script and supplying the root password and waiting for a lot of prerequisite software to be downloaded from the standard Linux Mint repositories. When that’s all finished, you’ll be prompted to reboot.

Once your PC is back up and running, you just launch Oracle’s own runInstaller as usual, click [Next] lots of times, and wait for the inevitable linking errors to occur once the installer tries to build Oracle binaries.

At that point, you just double-click the linking-error-fix.sh script that should be sitting on your desktop, click [Retry] in the Oracle installer …and wait for the installation and database creation process to complete. Update: There’s now a full-blown article documenting what you do, step by step.

No messing around with editing various obscure compiler files by hand: just run the first shell script to create the second; and run the second when the Oracle installer throws up its first linking error.

I’m running out of prime ministers these days, so this new script doesn’t have a fancy name. It’s just my “Oracle on Mint” script :-) It has been tested on all three of the Cinnamon, Mate and KDE versions of Linux Mint 17.2. There are no differences in how it runs on any of them.

Whether or not I do end up ditching Windows 10 for LM17.2, I can’t yet say: but being able to run Oracle on LM17.2 certainly makes the idea of a transition a whole lot more feasible.

When I’m done…

I hesitate to draw comparisons between me and Michaelangelo doing the Sistine Chapel, but I am reminded of Rex Harrison (ok, the Pope) forever asking Charlton Heston (ok, Michaelangelo): “When will you make an end of it!” And Heston,Michaelangelo tartly replying, “When I’m done!”

Well, we’re finished: the Wilson server article has just been completed, and only a week overdue :-)

And that completes the Churchill framework. Use it to its full and you now end up with two two-node clusters, one of which becomes a primary 2-node RAC, leaving the other to become a 2-node RAC running a standby copy of the primary; one network services server and its backup (Churchill and Attlee), and a beefy Enterprise Manager server to keep an eye on everything else (Wilson).

It’s quite a nice environment in which to try out things like patching, failover and switchover, configuring Cloud Control to send meaningful alerts, and so on. Happily, I’ve used 11.2.0.3, 11.2.0.4 and 12.1.0.2 with equally good results, and on top of CentOS 6.5 and 6.6, with an OEL 6.5 in there somewhere too.

My main desktop runs this complete infrastructure very nicely; my Toshiba (16GB RAM, 1.5TB solid state HDD) runs it just as nicely, though I have to dial down the memory numbers for my database VMs. My poor, ageing HP Folio 13 (8GB RAM, 256GB Solid State) has no problem with a 2-node RAC and an Enterprise Manager, but gettng it to do an additional 2 nodes for Active Data Guard practice is pushing it a bit. Maybe I should buy more solid state hard drive for it?! There’s a thought…

It’s everything I’ve ever wanted in a ‘build Oracle properly, routinely, accurately, automatically’ tool.

So now I can stop…

Today is an important day…

As the Ferryman in Benjamin Britten’s Opera “Curlew River” puts it, “Today is an important day”.

For today would have been Britten’s 101st birthday. Exactly one year ago today, I was settling down at the back of the Maltings Concert Hall, Snape for the Centenary concert (and a good one it was, too!) Twenty-six years ago, I was settling down in my seat at the Wigmore Hall for his 75th anniversary concert. And thus it has often been for more than half my life: today is spent playing pretty much nothing but Britten from dawn to dusk, and we pray that ToH thinks to do the vacuuming tomorrow rather than today!

Birthdays are for giving, of course (as I constantly have to remind ToH!) In this case, I’ve decided to release version 1.3 of Churchill, which has now been tested for 11.2.0.3, 11.2.0.4 and 12.1.0.2, for standalone, RAC, RAC+Data Guard and 12c Cloud Control installations. I’ve also taken the opportunity to tidy things up a lot, so necessary files are housed more appropriately, rather than all being plonked into a single directory. There are some more documentation issues that arise as a result of the clean-up, but those are relatively minor and should be done by tomorrow. Assuming I am not made to do the vacuuming tomorrow as penance…

Update 25th November: Beware of birthday gifts bought in a hurry! The new 1.3 ISO of Churchill was missing a key file (the ksh RPM), without which all attempts to run the root scripts at the end of a Grid Infrastructure install would fail. Oops. Now corrected (without incrementing the version number again: call it “1.3 Update 1” if you like… Microsoft can be such an inspiration!).

Gone

As promised, Salisbury and Asquith have been “retired” and have accordingly disappeared from the front page. They can still be reached from the Downloads page, though, should anyone still need them.

Churchill is now very nearly completely documented and replaces both. The only thing still missing is the description of how to create a Wilson server to act as an Enterprise Manager Cloud Control, and that should be finished by the end of the week.

I’ve also set up my own “ownCloud” hosting service and am hosting the Churchill ISO from there rather than from Dropbox. I think it’s all working and the necessary files are available to all, but if you run into any problems, just drop me a line and I’ll get it sorted ASAP.