Charles Wilson (pictured left) was a Scottish physicist in the early part of the 20th Century who invented the first cloud chamber. Particles ionizing the air in his chamber caused ‘vapour trails’ to form and you could thus “see” atomic particle interactions for the first time. It was no small achievement and he won the 1927 Nobel prize in physics for it. He also got a crater on the moon named after him (though a couple of other ‘Wilsons’ are involved there, it has to be said).

For the purposes of this blog, the inventor of the cloud chamber sounds like a good name to use for a new Asquith-framework component server that’s to be used as an Enterprise Manager 12c “Cloud Control” host.

A “Wilson server” has 4GB RAM, ~40GB hard drive and a dual-core CPU. It runs Oracle Database 11gR2, and that database gets used as the repository for an installation of Enterprise Manager 12c Cloud Control Update 3. Once built, you add ‘targets’ to Wilson’s Enterprise Manager (EM) and thus can use it to graphically monitor and control your other Asquith-based servers. You can, for example, use EM to create a Gamow-Dalton Data Guard copy of your original Alpher-Bethe RAC, entirely graphically.

So that’s what Wilson is. How do you build one? Well, you make sure your original Asquith server is built with the new 1.08 version of the Asquith ISO, which is available from the usual place. To upgrade a previously-built Asquith server to version 1.08, simply mount the 1.08 ISO then copy the asquith.ks, rosebery.ks and kickstart.php files over the top of the existing version of each file stored in the /var/www/html directory.

The updated Asquith Server also needs to house copies of the three files which make up the Enterprise Manager 12c Update 3 software stack. You can obtain all three required zip files from OTN. However, you have to re-name the downloads before they can be used. So, whilst you download files called em12103p1_linux64_disk1.zip, em12103p1_linux64_disk2.zip and em12103p1_linux64_disk3.zip you have to rename them, once they’re copied to Asquith’s /var/www/html directory to be:

  • oraem-12cU3-1of3.zip
  • oraem-12cU3-2of3.zip
  • oraem-12cU3-3of3.zip

If you source your download from somewhere other than OTN (say, from edelivery), the download files will have completely different names -but you must still rename them to conform to the names shown above.

You obviously also need your Asquith server to be hosting the Oracle Database software, too. Enterprise Manager software is a new addition to Asquith, not a substitute for any of its previous components.

So, once you’ve got a version 1.08 Asquith server hosting the Database and Enterprise Manager software, you’re ready to build your new Wilson server. You do so by booting it with appropriate CentOS/Scientific Linux/OEL boot media and creating a bootstrap line to initiate a Kickstart build. For Wilson servers, the bootstrap line will contain a new parameter: cloud=y.

If that parameter is present, then the Enterprise Manager software is copied off Asquith onto the new server’s /osource/em directory automatically, as part of the base build process. That’s in addition to the database software, which already gets copied to the /osource/database directory.

So, once your new server build is complete, you just have to:

  1. Log on remotely using ssh -X as oracle
  2. Launch the Oracle database installation procedure with /osource/database/runInstaller
  3. Modify the starter database by running /home/oracle/emprep.sh
  4. Launch the Oracle Enterprise Manager installation procedure with /osource/em/runInstaller

The full procedures to follow will be documented in due course, but the net effect is that it’s relatively painless to create a standalone EM server that can monitor, control and administer all your other Asquith component servers.

Note, finally, that I’ve added a new “speed key” to make building a Wilson server even simpler: sk=5. Specifying that implies your new component server will be called wilson.dizwell.home, have an IP address of, will be built using CentOS 6.5 and isn’t going to be running RAC or Data Guard. It will also imply “cloud=y”, so the EM software gets copied across and unpacked.

At its simplest, therefore, you add a dedicated Enterprise Manager server to your Asquith family of servers by using this bootstrap line when you start to build it:

ks= oraver=11201

(You can specify any other appropriate “oraver” there, of course: I use 11203 a lot these days, for example).

Finally, I’ll just mention that this feature isn’t coming to Salisbury any time soon. The ethos of Salisbury is command-line management. This is where Asquith and Salisbury really part ways, therefore.

Read the Instructions!

I have spent three days puzzling over this little nasty concerning the Enterprise Manager 12c Cloud Control (Release 3):

No matter how carefully I built my server, no matter what I re-configured, the EM installation would always fail at the point where it tries to create the MDS Schema. If you follow the trail of logs produced during the installation, you come across this:

2013-12-28 11:15:34.190 rcu:Extracting Statement from File Name: '/u01/app/oracle/oms12cr3/oms/rcu/integration/mds/sql/mds_user.sql'Line Number: 30
2013-12-28 11:15:34.190 rcu:Extracted SQL Statement: [CREATE USER &&1 IDENTIFIED BY &&2 DEFAULT TABLESPACE &&3 TEMPORARY TABLESPACE &&4]
2013-12-28 11:15:34.190 rcu:Statement Type: 'DDL Statement'
JDBC SQLException - ErrorCode: 922SQLState:42000 Message: ORA-00922: missing or invalid option

Error encountered executing SQL statement  FileName: '/u01/app/oracle/oms12cr3/oms/rcu/integration/mds/sql/mds_user.sql' LineNumber: '32'
java.sql.SQLSyntaxErrorException: ORA-00922: missing or invalid option

So, in a nutshell, creating the MDS schema failed because the Oracle database took exception to this SQL statement:


Now all those substitution variables take well-defined values: &&1 in this case is ‘SYSMAN_MDS’, &&3 is ‘MGMT_TABLESPACE’ and &&4 is ‘TEMP’. And knowing what password I suggested during the earlier phases of the installation, I know that &&2 is going to take a value of Oracle1!.

Why that value? Because I remember reading on one of the Enterprise Manager installer screens that the administrator, node manager and MDS passwords needed to be at least 8 characters long, start with a a letter, have at least one numeric and contain at least one special character. Which is, of course, pretty standard fare these days for complex passwords. So “Oracle1!” it was: a capital letter, a numeric, a special character, 8 characters long. What could go wrong?

Well, judging from the ORA-00922 error displayed in that EM installer log, quite a lot! And sure enough, if you take all the known values for all those substitution variables and plug them into a basic SQL statement, you can generate ORA-00922 errors on demand:

SQL> create user sysman_mds identified by Oracle1! 
  2  default tablespace mgmt_tablespace
  3  temporary tablespace temp;
create user sysman_mds identified by Oracle1!
ERROR at line 1:
ORA-00922: missing or invalid option

The problem, as the precise position of that asterisk makes plain, is the exclamation mark: Oracle doesn’t like special characters much… not unless you wrap them in double quotes. Had that SQL statement read create user sysman_mds identified by “Oracle1!”…, all would have been fine. But quote-less, it fails.

Clearly, this is an Enterprise Manager bug!! If it’s going to demand that you supply a password of sufficient complexity that it must contain a special character, then it damned-well ought to be able to submit it wrapped in quotes when its JDBC process try to submit “create user” statements to the database!!!

Well, maybe. But let’s just read the installer’s actual instructions again, shall we? (Something I only did, as I say, after three days of faffing around with this problem!):

Just focus on that error message at the bottom of the screen again. “Specify Administrator Password: Password must contain at least 8 characters, begin with a letter and include at least one numeric value”.

Er, where is the bit about ‘special characters’? Not there, that’s where. That, it turns out, was a figment of my imagination -no doubt tainted by over-familiarity with Oracle’s “password_verify” function which you can use as a database profile limit to enforce database password complexity checks and which does specify the use of punctuation characters. Clearly, what Oracle deems to be a minimum level of password complexity for its database is not quite the same thing as what it seeks to enforce as password complexity for its Enterprise Manager tool. :-(

In short, I had assumed the need to specify a punctuation or special character in the EM administrative password when, if I’d read the on-screen instructions carefully enough, no such special character is actually needed. Which is particularly unfortunate when the inadvertent use of a special character actually breaks the EM Installer!

So, two lessons here. One, always read the instructions printed on the tin. If it doesn’t demand a special character, don’t assume it does. And Two: Oracle’s EM Installer ought to be able to handle complex passwords and can’t. So that’s a bug in my book.

Oh, and Three: don’t use punctuation characters when specifying your EM 12c Cloud Control administrative password. It breaks Oracle, big- (or bug-) time.

2013 Wrap-Up

I started the New Year at a new employer, doing RAC and Data Guard configuration/setup and various bits of database development work. I end it with us preparing (finally!) for a disaster recovery test in which we will find out if I am as good as configuring multi-node Data Guard setups as I think I am! First few weeks of the New Year will reveal all…

Meantime, my job’s changed a bit over the course of the year. Day-to-day management of the database infrastructure having been (or about to be) outsourced to a consultancy, I am more focussed than ever on development activities… and these have suddenly taken a turn in the direction of SAP BusinesObjects. Which makes me Universe Designer-in-Chief, trying to master a product I’ve never really used in anger before. Some learning curve! Fun times ahead, methinks.

Lucretia (the cat) has proved as young-at-heart as ever, despite turning 18. She lost a lot of weight at the veterinary ‘hotel’ we put her in whenever we take a holiday. So much so that, this time, I thought we were going to lose her. She literally couldn’t purr right: she’s normally double-bass profundo but could only manage a pathetic sort of high-pitched wheeze on our return. If she’d eaten at all during the fortnight we were on holiday, it would only have been on day 1. But four days after our return, having not touched food no matter how nicely we put it out or varied it, she finally got her mojo back. The purr returned on day 6. And now we’ve been back two weeks, I doubt you’d notice the problem. She’s good for another year or two after all, it seems.

Chandler (the wallaby) still looks as if she’ll be around for years, to, where “around” can mean “eating out of the cat’s bowl” or “watching TV in the lounge” as she deems appropriate. Four other wallabies have persistently attended the back door throughout the year, making us feel very lucky and very happy.

Personally, I spent a lot of the year gearing up for the Benjamin Britten centenary. We got the Royal Mail stamps, then the Royal Mint 50p proof coin, then the full scores of the War Requiem and Gloriana. Then it was a long, long wait for the flight to Blighty, spent for the most part panicking that I’d fall prey to some virus or bug or other. But in the end, I awoke in Aldeburgh on the appointed day with no sniffles and just the merest hint of a chest infection that has subsequently developed into a nice dose of body-wrenching bronchitis. Returning to the Red House (where Britten and his partner Peter Pears lived from the 1950s to the 1970s) for the first time since 1986 was one of those truly memorable, once-in-a-lifetime, experiences.

“Have you ever been here before” asked the guide. “Why yes, actually,” I replied. “In 1986, I came and had a cup of tea and a digestive biscuit with Sir Peter Pears, who’d invited me to discuss the works of Britten”. At which the guides (pleasingly!) expressed suitable amounts of awe and admiration. I didn’t remember the room where he’d handed me the digestive, because they’ve moved everything around, back to the way it was in 1967. But, satisfyingly, the dip in the floor where I embarrassedly tripped over in 1986 was still there, still familiar… though somewhat less embarrassing second-time-around!

I have to say, “Britten Country” (that is, Aldeburgh, Snape, Orford, Thorpeness and their surrounds in east Suffolk) is utterly wonderful (though we were lucky with the weather: typical bright-sun-clear-sky-cool-temperatures November, basically). I’ve dreamt about it ever since returning. If someone would kindly donate about half a million pounds, I’d be buying a house there and moving back almost immediately. It was that good -and much better than I had remembered from my last trip there in 1989. God’s own country, for sure.

I only changed operating systems three times in the year, which is some sort of record (low!) for me. Windows 8 (and 8.1) were good, but uninteresting. Fedora 19 was good, but Gnome Shell is bloody awful, despite copius quanitites of goodwill and patience on my part. And Linux Mint KDE 16 is new and experimental, but I’m enjoying it. No doubt, the story will develop in 2014…

Coming back from Paris a few weeks ago was not the nicest flight I’ve ever had: the incipient chest infection finally took hold, and by the time we started the final leg from Kuala Lumpur, I expect I was that sort of unavoidable, irritating passenger everyone dreads: my cough was persistent, loud and annoying. I end the year on a course of antibiotics as a result, though they don’t seem to be doing an awful lot.

Somewhat more worrying was the perpetual numbness in parts of my assorted limbs on the right-hand side of my body. A CT scan was ordered… and bizarrely tended to suggest I’d suffered a minor stroke at some point. Thus a referral for a more detailed MRI scan, which took place at the start of this week. If you haven’t had an MRI scan, I wouldn’t recommend one! The procedure consists of being inserted a very long way into a very confined space and being told to stay absolutely still for about half an hour (which is much, much harder than it sounds!). They also tell you that ‘the machine will make lots of noises’… and you sort of expect a bit of a hum, whirr and kerphhutt. Actually what you experience, despite ear plugs AND ear defenders, is the most godawful, body-ripping WHOOOOOOOOOORRRRRRRRRRRR-KERPPPPPPPPPPPPPPPPPPHGHGGHHHHHHHT-BOOOM-ditty-BOOOOM-ddiiittty-kerdiddle-kerdiddle-bludge-phhot-quot-kerpling. Repeatedly, incessantly and at a volume I can only imagine that, without ear defenders, would make your eyes bleed.

I’m not surprised to be told that approximately 50% of people who start an MRI scan hit the emergency release button they give you and thus fail to complete it. I was pretty close to hitting ’emergency stop’ myself, but managed to make it through, with gritted teeth.

I am dubious about what passes for much of ‘medical science’ these days: doctors taking a quick look in the back of your throat, making three passes of your chest with a stethescope and then reaching for the day-off-work sick-form pad doesn’t actually count, in my view, as medicine. MRI machines, on the other hand, really are cutting-edge science and definitely count as “proper medicine”… but they really need to do something about how user-unfriendly they’ve made it!

Anyway, I haven’t had the results yet, but the numbness persists. We shall see. I have no symptoms apart from being able to stick needles into parts of my right leg without it hurting, when it probably should. Oh, and the tip of my nose constantly feels cold, too, which is an odd sensation to be sure.

Otherwise, the year has not been at all bad. I’ve enjoyed work and the company of many colleagues, I’ve loved getting back to Cambridge and Aldeburgh, ToH is happy, the cats and wallabies are content, I’ve spent more than I should on music and less than I ought on computer hardware (not something that has happened often in the past decade or so, I can tell you!). My sisters back in England are all well, as are my various nephews and nieces. I turn 50 on January 7th… all-in-all, it’s not been a bad run-up to it.

I shan’t be posting much (if anything) between now and the turn of the New Year, so can I take this opportunity to thank all my readers and correspondents for their continued interest in all things Dizwell… and to wish you all a joyous Christmas and a very happy New Year.

Mint, 11g and KDE

For no particular reason, I decided I really didn’t like Gnome Shell as implemented by Fedora 19. As with previous Fedora installations, I thought I did for a while, but soon came to the realisation that it just gets in the way of doing things efficiently. I didn’t fancy installing MATE (flakey) or Cinnamon (a hack) desktops either.

So I did something I haven’t done for years and years: installed a KDE desktop. Much to my surprise, I really liked it. I’m now wondering what it was about KDE4 desktops in the past that put me off… but I can’t remember. All I can say is that whatever it was, it’s no longer an obstacle. Instead, I find KDE a pleasure to use, highly intuitive and aesthetically pleasing. For anyone transitioning to Linux from Windows, it’s a much more familiar GUI than Gnome Shell is, too.

Not only did I switch to KDE, but I also switched from Fedora to Linux Mint (the release candidate for the KDE spin, version 16, called “Petra” is available now and will eventually upgrade over time to the final release when it’s ready in a couple of weeks’ time). I’ve used Mint before (I’ve even donated money to the project, making it only the second version of Linux I’ve personally paid money for), and I’ve always found it a pleasure: lots of software in the repositories; applications such as Handbrake, Stellarium, VMware Workstation and others… all easy to install; all major multimedia codecs pre-installed from the get-go; and so on. Mint 16 KDE doesn’t disappoint on any of these grounds, and is definitely ‘fit for purpose’.

Getting Oracle installed on it was a bit tough, but I used my existing Gladstone script and tweaked it a bit and all went well in the end:

I may document that more fully… perhaps even release an updated version of Gladstone to automate the various fix-up steps I had to take. But not right away (certainly not while it’s still in release candidate mode, anyway!)

Anyway, in the meantime: Linux Mint 16 “Petra” KDE Release Candidate …recommended.

Asquith and the new Red Hat

Whilst I was busy planning my Paris perambulations, Red Hat went and released version 6.5 of their Enterprise Server distro. Oracle swiftly followed …and, even more remarkably, CentOS managed to be almost equally as swift, releasing their 6.5 version on December 1st. Scientific Linux has not yet joined this particular party, but I assume it won’t be long before they do.

I had also assumed Asquith would work unchanged with the new distro -but I hadn’t banked on the clumsy way I originally determined the distro version number which actually meant it all fell into a nasty heap of broken. Happily, it only took a minute or so to work out which bit of my crumbly code was responsible and that’s now been fixed.

Asquith therefore has been bumped to a new 1.07 release, making it entirely compatible with any 6.5 Red Hat-a-like distro (and any future 6.x releases, come to that).

Another feature of this release is that the ‘speedkeys’ parameters have been altered so that they assume the use of version 6.5 of the relevant distro. That is, if you build your RAC nodes by using a bootstrap line that reads something like ks=…, then you’ll be assumed to be using a 6.5 distro and the source OS for the new server will be assumed to reside in a <distro>/65 directory.

If you want to continue using 6.4 or 6.3 versions, of course, you can still spell that out (ks=…). You just can’t use speedkeys to do it.

An equivalent update to Salisbury has also just been released.