Finished

Finished! And only three weeks late!

I’m referring to Churchill 1.7, a major overhaul of the Churchill framework and its accompanying documentation, making it work on RHCSL 6.8, for 12c only, with Enterprise Manager Cloud Control 13c and a complete overhaul of the bootstrap options available when kickstarting O/S installations.

At some point toward the end of January, it morphed into practically a complete re-write… and I thought seriously about calling it quits and declaring it to be version 2.0. But I’ve stuck with the incremental versioning for now. (I’ve been saving version 2 for when I get round to making it work with RHCSL 7.x distros).

I’m finished in another sense, though, too: the contract to purchase a house in Nottingham is ready to sign and it accordingly looks very much as though I’ll be becoming an ex-Aussie (or a re-Englishman, I suppose, depending on your point of view) on or around 6th March. I may not have much time to post much here given the packing, flight-booking, passport-checking, Internet banking, etc etc shenanigans that now ensue. If I can I will, but otherwise I’ll be back online toward the end of March, live from Nottingham 🙂

Churchill Changes

I’ve just published version 1.7 of Churchill. It contains a lot of changes.

The main one is that I’ve removed a few dependencies on .i686 packages. That means the O/S installations can now all take place off the first DVD alone. No second DVD is prompted for, in other words.

This in turn brings about the biggest single benefit of the new release: it works with CentOS 6.8 (and Scientific Linux 6.8, too).

In fact, Churchill 1.7 now reverts to making CentOS 6.8 the default O/S assumed to be in use. (You can still always specify ‘redhat’, ‘sl’ or ‘oel’ if you prefer to use real Red Hat, Scientific Linux or Oracle Enterprise Linux, of course; and you can always specify an earlier distro version if you prefer to stick with (say) 6.3 -though I can’t think why you’d particularly want to).

The other big change is that the bootstrap lines are now trivially easy. Back in 2013 when I first released Churchill, it seemed like a good idea to make it as flexible as possible so that users could specify their own IP addresses and hostnames; but this just made for really lengthy bootstrap lines and confused the heck out of everybody!

So, the simplify brush has been daubed all over Churchill. You now must use the speed keys 1 to 4 as you build your nodes (or you can instead specify their corresponding hostnames). By specifying the speed key or hostname, you automatically define all the pesky details about IP address and interconnect IP address. It makes things a lot simpler and less confusing, I think. It also makes it a bit less flexible… but that’s the price you pay for simplicity. Ask the Gnome developers!

Other changes flow in consequence: the filecopy=y/n parameter is no longer required. If you are building nodes 1 or 3, file copying is assumed to be ‘yes’; if you are building nodes 2 or 4, it’s assumed to be ‘no’. Likewise, there’s no dg=y/n parameter any more: if you are building nodes 1 or 2, it’s assumed to be ‘no’; but build nodes 3 or 4, it’s assumed to be ‘yes’.

It is still possible to say sk=1&rac=n (or hostname=alpher&rac=n), though: that’s if you are building node 1 but want to use it in standalone mode.

As with the previous release (1.6), Churchill only works to build 12c standalone and RAC/Data Guard environments: there is no Oracle 11g support these days.

These are substantial changes and mean Churchill now works in ways quite different from before. That obviously affects the way it is documented. Those documentation changes are being made as I write and should be ‘live’ by the time you read this. Since the old versions of Churchill are kept available on the old site, I’ll keep the original documentation available on that old site too, at least for now.

On this site, however, only the 1.7 version will be available and documented.

State of Play

churchill150It has been almost a year since I did anything to my Churchill automation framework. (If you didn’t know or realise, Churchill lets you easily create virtual multi-node RAC and Active Data Guard environments).

I took a look at it lately to see about refreshing it. I discovered that whilst Churchill’s speed keys will assume that you’re using CentOS 6.6, CentOS has actually released versions 6.7 and 6.8 since the last Churchill refresh (as have Red Hat, Scientific Linux and Oracle, of course)! It is also still the case that no version of Churchill works with a RHCSL 7.x O/S (because systemd screws up network automation).

No worries, I thought to myself: tweak a few things and at least bring Churchill up-to-date-ish with version 6.8 as the new speedkey default.

No chance!

In their wisdom, the CentOS developers decided to split the main DVD release of 6.8 over two DVDs. That means Churchill’s trick of copying its own installation media onto its own hard disks (for subsequent automated network installs onto the RAC nodes to use) fails, because it can only copy one of the two install DVDs. (There is a special, 6.8-only Dual Layer DVD you can use, and that would work… but I decided not to go there for now).

This, combined with the systemd debacle, basically means that Churchill is dead in the water, and I therefore have made only one more tweak to it -whereby Scientific Linux version 6.7 is assumed to be the speedkey default- and after that, I won’t be maintaining it further.

Something new is required to automate 12c RACs on RHCSL 7.x environments… and I don’t know what that will be yet! Watch this space, I guess…

When I’m done…

I hesitate to draw comparisons between me and Michaelangelo doing the Sistine Chapel, but I am reminded of Rex Harrison (ok, the Pope) forever asking Charlton Heston (ok, Michaelangelo): “When will you make an end of it!” And Heston,Michaelangelo tartly replying, “When I’m done!”

Well, we’re finished: the Wilson server article has just been completed, and only a week overdue :-)

And that completes the Churchill framework. Use it to its full and you now end up with two two-node clusters, one of which becomes a primary 2-node RAC, leaving the other to become a 2-node RAC running a standby copy of the primary; one network services server and its backup (Churchill and Attlee), and a beefy Enterprise Manager server to keep an eye on everything else (Wilson).

It’s quite a nice environment in which to try out things like patching, failover and switchover, configuring Cloud Control to send meaningful alerts, and so on. Happily, I’ve used 11.2.0.3, 11.2.0.4 and 12.1.0.2 with equally good results, and on top of CentOS 6.5 and 6.6, with an OEL 6.5 in there somewhere too.

My main desktop runs this complete infrastructure very nicely; my Toshiba (16GB RAM, 1.5TB solid state HDD) runs it just as nicely, though I have to dial down the memory numbers for my database VMs. My poor, ageing HP Folio 13 (8GB RAM, 256GB Solid State) has no problem with a 2-node RAC and an Enterprise Manager, but gettng it to do an additional 2 nodes for Active Data Guard practice is pushing it a bit. Maybe I should buy more solid state hard drive for it?! There’s a thought…

It’s everything I’ve ever wanted in a ‘build Oracle properly, routinely, accurately, automatically’ tool.

So now I can stop…

Today is an important day…

As the Ferryman in Benjamin Britten’s Opera “Curlew River” puts it, “Today is an important day”.

For today would have been Britten’s 101st birthday. Exactly one year ago today, I was settling down at the back of the Maltings Concert Hall, Snape for the Centenary concert (and a good one it was, too!) Twenty-six years ago, I was settling down in my seat at the Wigmore Hall for his 75th anniversary concert. And thus it has often been for more than half my life: today is spent playing pretty much nothing but Britten from dawn to dusk, and we pray that ToH thinks to do the vacuuming tomorrow rather than today!

Birthdays are for giving, of course (as I constantly have to remind ToH!) In this case, I’ve decided to release version 1.3 of Churchill, which has now been tested for 11.2.0.3, 11.2.0.4 and 12.1.0.2, for standalone, RAC, RAC+Data Guard and 12c Cloud Control installations. I’ve also taken the opportunity to tidy things up a lot, so necessary files are housed more appropriately, rather than all being plonked into a single directory. There are some more documentation issues that arise as a result of the clean-up, but those are relatively minor and should be done by tomorrow. Assuming I am not made to do the vacuuming tomorrow as penance…

Update 25th November: Beware of birthday gifts bought in a hurry! The new 1.3 ISO of Churchill was missing a key file (the ksh RPM), without which all attempts to run the root scripts at the end of a Grid Infrastructure install would fail. Oops. Now corrected (without incrementing the version number again: call it “1.3 Update 1” if you like… Microsoft can be such an inspiration!).

Gone

As promised, Salisbury and Asquith have been “retired” and have accordingly disappeared from the front page. They can still be reached from the Downloads page, though, should anyone still need them.

Churchill is now very nearly completely documented and replaces both. The only thing still missing is the description of how to create a Wilson server to act as an Enterprise Manager Cloud Control, and that should be finished by the end of the week.

I’ve also set up my own “ownCloud” hosting service and am hosting the Churchill ISO from there rather than from Dropbox. I think it’s all working and the necessary files are available to all, but if you run into any problems, just drop me a line and I’ll get it sorted ASAP.

Salisbury and Asquith, my ‘frameworks’ for automated, nearly-hands-free building of Oracle servers, are retiring. Which is to say, I’m not going to maintain them any more.

My attempts over the years to persuade my System Admin colleagues at work that RAC via NFS (as Salisbury uses) might be a good idea have all fallen on deaf ears, Kevin Closson’s fine articles on the subject notwithstanding. So Salisbury became a bit of a dead end after that, which is why I cooked up Asquith. Asquith uses real iSCSI (as real as anything a virtual environment can cook up, anyway!) and layers ASM on top of that and thus provided me with a playground that much more faithfully reflects what we do in our production environment.

But it’s a pain having two frameworks doing pretty much the same job. So now I’m phasing them out and replacing them with Churchill. The Churchill framework uses NFS (because it’s much easier to automate the configuration of that than it is of iSCSI), but it then creates fake hard disks in the NFS shares and layers ASM on top of the fake hard disks. So you end up with a RAC that uses ASM, but without the convoluted configuration previously needed.

The other thing we do in production at work is split the ownership of the Grid Infrastructure and the Database (I don’t know why they decided to do that: it was before my time. The thing is all administered by one person -me!- anyway, so the split ownership is just an annoyance as far as I’m concerned). Since I’ve been bitten on at least one significant occasion where mono- and dual-ownership models do things differently, I thought I might as well make Churchill dual-ownership aware. You don’t have to do it that way: Churchill will let you build a RAC with everything owned by ‘oracle’ if you like. But it does it by default, so you end up with ‘grid’ and ‘oracle’ users owning different bits of the cluster, unless you demand otherwise.

Other minor changes from Asquith/Salisbury: Churchill doesn’t do Oracle 11.2.0.1 installations, since that version’s now well past support. You can build Churchill infrastructures with 11.2.0.3, 11.2.0.4 and 12.1.0.2. Of those, only the last is freely available from otn.oracle.com.

Additionally, the bootstrap lines have changed a little. You now invoke Alpher/Bethe installations by a reference to “ks.php” instead of “kickstart.php” (I don’t like typing much!). And there’s a new bootstrap parameter: “split=y” or “split=n”. That turns on or off the split ownership model I mentioned earlier. By default, “split” will be “y”.

Finally, I’ve made the whole thing about 5 times smaller than before by the simple expedient of removing the Webmin web-based system administration tool from the ISO download. I thought it was a good idea at the time to include it for Asquith and Salisbury but, in fact, I’ve never subsequently used it and it made the framework ISO downloads much bigger than they needed to be. Cost/benefit wasn’t difficult to do: Webmin is gone (you can always obtain it yourself and add it to your servers by hand, of course).

The thing works and is ready for download right now. However, it will take me quite some time to write up the various articles and so on, so bear with me on that score. All the documentation, as it gets written, will be accessible from here.

The short version, though, is you can build a 2-node RAC and a 2-node Active data guard with six basic commands:

  • ks=hd:sr1/churchill.ks (to build the Churchill Server)
  • ks=http://192.168.8.250/ks.php?sk=1 oraver=11203 ksdevice=eth0 (to build Alpher)
  • ks=http://192.168.8.250/ks.php?sk=2 oraver=11203 filecopy=n ksdevice=eth0 (to build Bethe)
  • ks=http://192.168.8.250/atlee.ks (to build Atlee, the file server for the Data Guard nodes)
  • ks=http://192.168.8.250/ks.php?sk=3 oraver=11203 ksdevice=eth0 (to build Gammow)
  • ks=http://192.168.8.250/ks.php?sk=4 oraver=11203 filecopy=n ksdevice=eth0 (to build Dalton)

With Churchill and the rest of the crew, I can now build a pretty faithful replica of my production environment in around 2 hours. Not bad.

Salisbury and Asquith will remain available from the front page until the Churchill documentation is complete; after that, they’ll disappear from the front page but remain available for download from the Downloads page, should anyone still want them.