Dizwell Informatics

News from Nowhere

Build Wilson (Enterprise Manager 13c)

1.0 Introduction

A Wilson server is what the Churchill framework calls its Enterprise Manager server: the ‘watch box’ that looks over all the other framework servers and tells you what they’re running and how they’re doing -and gives you all the tools needed to manage them with a simple point-and-click interface.

It’s the new, centralized way of managing database (and application server) infrastructures, as opposed to the old-fashioned way of logging on to each server in turn and issuing SQL or O/S commands!

To be honest, Wilson is complete over-kill for the Churchill framework: there just aren’t enough servers to be managed for it to be a worthwhile investment of resources (and boy: are you going to need some resources!). But large-scale enterprises will definitely be using Enterprise Manager to do serious work, so it’s as well to be familiar with it and its capabilities, no matter how inadequate the testing ground you run it in at home might be.

2.0 Hardware Resources

To be frank about it, Wilson is a resource hog to out-hog all others. It requires large memory and hard disk allocations, because it does so much! Not only is there a complete Oracle 12c database running on it (to act as the ‘repository database’ for the Enterprise Manager application, but there’s also a complete Weblogic application server too -and Weblogic is not known for being slimline!

So you’re going to need a beefy virtual machine:

  • at least 2 x CPU
  • 80GB hard disk
  • 8GB RAM
  • 1 x CD-ROM
  • 1 host-only network interface (connected to the same network interface that Churchill itself uses).

If you’re building Wilson after you’ve built a 2-node Alpher/Bethe RAC, you are now looking at memory requirements in total of 0.5GB for Churchill + 10GB for Alpher and Bethe combined + 8GB now for Wilson. That’s 18.5GB in total… which has therefore blown the budget for a 16GB laptop. You could of course use Wilson in a purely Churchill+Alpher-as-a-standalone-server environment -that’s just 13.5GB and might be possible on a 16GB laptop.

But my central message is: Wilson is challenging and demanding and definitely something for a beefy desktop and not a cutesy netbook.

3.0 Constructing the VM

As was just mentioned, your Wilson VM needs to be substantial -and networked using the same host-only network subnet that Churchill uses.

In VirtualBox, for example, you would have created a host-only network interface called vboxnet0 when your first created Churchill and a second called vboxnet1 when you built Alpher and Bethe as RAC nodes. It is the vboxnet0 interface your Wilson server needs to use now -the public one that all Churchill framework servers share to talk amongst themselves and not the private one that RAC nodes use to perform cache fusion transfers of data between Oracle instances.

Here is my VirtualBox Wilson:

Note the large 80GB hard disk: it’s dynamically-sized, so it might not actually get that big. But I’d count on it getting a substantial chunk of the way there! Note too the 8192MB RAM allocated: Weblogic will eat all of that without pausing to think about it! Finally, as ever in the Churchill framework, note that I am proposing to boot my Wilson server with a CentOS 6.8 ISO -because that’s what I built my Churchill server with. You cannot mix and match distros or versions, so if your Churchill uses OEL 6.4 (as an example), then Wilson must be built using that too.

4.0 A Word about Software

It was mentioned back when I discussed building the Churchill server, but in case you missed it… Wilson will require a huge quantity of software to be built properly -and all of it needs to be pre-loaded into the Churchill server’s /var/www/html/oracle directory:

Down the bottom of this listing, you’ll see the files with which you’re probably very familiar: the oradb- and oragrid- files which make up the standard Grid and Database 12c installation zip files, though they have been renamed to conform with Churchill’s requirements. Wilson runs an Oracle 12c database to make Cloud Control work, so it needs those files as much as Alpher or Bethe did (though, in fact, the database is going to run on a traditional file system so doesn’t actually need the Grid Infrastructure files).

The top seven files are the ones needed to build the Weblogic application server which is at the heart of Cloud Control. There are five zip files (numbered 2 to 6, just to keep you on your toes!), one bin file and a ‘database template’ file. All can be obtained from OTN for free, as part of the 13.2.0.0 release of Enterprise Manager Cloud Control. None of those files have been renamed -and it’s important that you don’t, because when you run the .bin file, it will expect to find the ’em13200′ zip files with the right names in the right places.

That’s quite a software collection: 13GB+ of software. It’s the reason Churchill needed a 75GB hard disk to start with; it’s also why Wilson needs to be built with an 80GB one!

5.0 Booting Wilson

Building Wilson is, as ever in the Churchill framework, a matter of powering up the new VM, hitting the <TAB> key and appending a bootstrap line to the end of the text that’s already there. For Wilson, the special bit of magic required in that bootstrap line is to either use speed key 5 or to specify a hostname of ‘wilson’ (the two things are exactly equivalent to each other).

On CentOS 6.8, for example, I’d type this:

I could also have typed ks=http://churchill/ks.php?hostname=wilson, of course.

If you are not using CentOS 6.8 as your Churchill framework O/S, you will need to add ‘distro=’ and ‘version=’ parameters to that string. For example, if you were running Red Hat 6.9, you’d type:

ks=http://churchill/ks.php?sk=5&distro=rhel&version=69

As always, the exact ordering of the parameters is irrelevant, and the version value is your actual O/S’s value stripped of its decimal point (so 6.3 becomes 63, 6.4 becomes 64 and so on).

Wilson is unique in the Churchill framework in that it is NOT going to be using ASM for its database storage -and therefore there is no Grid Infrastructure install to be performed. There is therefore no concept of ‘split ownership’ and the parameter “split=y” or “split=n” therefore is not used to build Wilson (if you specify it anyway, it is simply ignored).

When you’ve typed your bootstap line, press [Enter] to initiate the O/S installation. It will be completely automated and you won’t need to do anything else until you are prompted to reboot at the very end of it all. After that, Wilson will come back up as the text-only server you’ve come by now to expect in the Churchill framework!

6.0 What happens next -the overview

Once Wilson has been rebooted, you’re going to be spending a lot of time doing three separate operations:

  1. A software-only install of the standard Oracle 12c database software
  2. Separately creating a database that can act as a Cloud Control repository
  3. Installing Cloud Control itself

When building earlier Churchill framework servers, we usually accept the offer to create a database whilst installing the Oracle 12c database software -but this is not appropriate for Wilson. Cloud Control stores all its configuration details inside a 12c database, for sure; but it needs to be a database with special characteristics, storage capabilities and user accounts. The standard templates provided by Oracle’s database software don’t allow us to build such a beast, which is why you have to have previously downloaded (onto Churchill) a special database template; and why we have to create the database as a separate exercise from the software installation.

I will warn you in advance, too: the Cloud Control installation takes at least an hour. If you’re running on feebler hardware than I’m lucky to be using, expect it to take longer still!

Taking each of those steps in turn, therefore…

7.0 Installing Oracle 12c

As ever with Churchill framework installs, the server you’re dealing with is a text-only affair… which is a challenge when all the Oracle work you have to perform on it requires a graphical, GUI interface!

We’ve met this challenge previously, however: you simply need to connect to Wilson from a PC that is capable of displaying GUI interfaces, using a remote X connection with X-forwarding enabled. I discussed how to do this in some detail in the standalone Alpher article. The short version for here is, if your physical desktop is running Windows, use Mobaxterm; if it’s running Linux, use ssh -X connections.

From my physical desktop, therefore, I simply issue this command in a terminal window:

ssh -X oracle@192.168.8.252

The user account being used there is mandated: the owner of everything we install on Wilson is an account called ‘oracle’ and there’s no variation on that allowed. The IP address is also mandated by the automated installation process facilitated by Churchill: 192.168.8.252 is Wilson’s IP address and is not negotiable. Finally, the “-X” switch is what makes applications running on Wilson able to draw their screens on your physical PC’s desktop using the X protocol, so that’s not optional either!

As is usually the case, the password for the oracle user is oracle.

As is also always the case the first time you connect to a server via ssh, you’ll be asked to accept the security credentials for the new machine before you get to supply the password for the oracle user:

Now you are ready to start launching the Oracle software installation by typing the command:

/osource/database/runInstaller

The usual splash-screen appears and disappears and leaves you sitting at this:

There is nothing very exciting about the software installation -by the time you build Wilson in the Churchill framework, you’ve probably performed enough of them to know what’s coming! Nevertheless, here’s a slideshow to step you through the process:

To re-emphasise: it’s important you make this a software-only installation. For the rest, the defaults usually apply without the need to change any of them.

8.0 Creating a Repository Database

Now your Oracle software has been installed successfully, you are ready to create the database which Cloud Control will store its configuration data in -called the cloud control repository. That database is built from a template (which you will have downloaded to Churchill before even starting the construction of Wilson).

Logged on as the oracle user, therefore, you now type the following commands:

cd $ORACLE_HOME/assistants/dbca/templates
unzip /osource/em/dbtemplate.zip

The output you’ll see should resemble this:

oracle@wilson:~ [emrep]$ cd $ORACLE_HOME/assistants/dbca/templates
oracle@wilson:templates [emrep]$ unzip /osource/em/dbtemplate.zip
Archive: /osource/em/dbtemplate.zip
 inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Large_deployment.sql 
 inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Medium_deployment.sql 
 inflating: set_repo_param_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0_Small_deployment.sql 
 inflating: shpool_12.1.0.2.0_Database_SQL_for_EM13_2_0_0_0.sql 
 inflating: 12.1.0.2.0_Database_Template_for_EM13_2_0_0_0_Large_deployment.dbc 
 inflating: 12.1.0.2.0_Database_Template_for_EM13_2_0_0_0_Medium_deployment.dbc 
 inflating: 12.1.0.2.0_Database_Template_for_EM13_2_0_0_0_Small_deployment.dbc 
 inflating: 12.1.0.2.0_Database_Template_for_EM13_2_0_0_0.dfb 
 inflating: 12.1.0.2.0_Database_Template_for_EM13_2_0_0_0.ctl

(In case you were wondering, Churchill renamed the template zip file from its original long-winded name to the simpler ‘dbtemplate.zip’ as part of its automatic build process when Wilson was first being constructed. I should also mention that, technically, there is no need to use one of these templates: an ‘ordinary’ database can be converted for Cloud Control’s use. But it’s quicker and more reliable to use an Oracle-supplied template, as you might well expect).

Once the templates have been unpacked, you can launch the Database Creation Assistant (dbca) from your physical desktop, still using a remote-X session (i.e., ssh -X oracle@192.168.8.252). Just type the command:

dbca

As usual, I’ll step you through the wizard that now appears, as follows:

Once you have a functioning database successfully created, you’re ready to move on to the third and last step in creating Wilson: installing Cloud Control.

9.0 Installing Cloud Control

You need to be connected to Wilson as the oracle user, over an SSH connection on which X-forwarding has been enabled -so that again means connecting from your physical desktop using the command:

ssh -X oracle@192.168.8.252

…if you aren’t already so connected.

You kick off the Cloud Control installation process by issuing the following commands:

cd /osource/em
chmod +x em13200_linux64.bin
./em13200_linux64.bin

To start with, not much will appear to happen except that a lot of dots will appear in the terminal session you’re using:

That’s the Cloud Control binary unzipping the other ’em13200′ files you see listed in that screenshot. It’s why you cannot muck around with the file names making up the Cloud Control software set: the binary expects to find files with their ‘right’ names! Anyway, as you can see, there are a lot of zip files to unzip, so those dots will appear for quite a while, until eventually… you’ll see this:

This is the start of the Cloud Control installation wizard -and for the most part, you can just press [Next] to step your way through it. But, as usual, here’s a slideshow showing you the entire process:

Once you get that far, you can open a browser on your physical PC and attempt to connect to the Cloud Control URL mentioned in the last part of the installation. Where you were told to connect to ‘wilson.dizwell.home’, however, you’ll need to translate that into the IP address of that VM -192.168.8.252. The correct URL to connect to will therefore be https://192.168.8.252:7802/em. Note that this is an https address, not a plain http one.

You will get the usual error at this point about the connection not actually being secure:

Use your browser’s ‘advanced’ techniques to connect to the URL regardless. In Firefox, that’s Advanced -> Add Exception -> Confirm Security Exception. In Chrome, it’s Advanced -> Proceed. Whatever browser you use, you should end up looking at this:

You are not logging onto a database at this point, but into Cloud Control itself, so the username to be used is sysman and the password is the complex one you were forced to provide during the Cloud Control installation process.

The first time you log onto Cloud Control, you get asked a couple of questions:

If you need accessibility enhancements, switch them on and save; if you are lucky enough not to need them, then just leave them all switched off and click save; or you can click ‘deal with it later’, in which case you’ll be prompted to answer again next time you log in. Answer and click as you need. Next:

Inevitably, there’s a license agreement to sign up to. This is Oracle, after all. Be warned that this is not lightly clicked through in an actual production environment: there are cost implications for using any of this stuff in a real environment. Using at home for self-learning purposes, however, is fine. So you can click ‘I accept’ with a clear conscience.

Finally:

You are presented with a whole raft of possible home page templates to use. Probably, as a DBA, you’d click the ‘Databases’ option -but Cloud Control can manage a lot more than just databases, and other users might prefer different home pages. It doesn’t hugely matter what you pick now, since you can always change your selection later. But to kick things off, click the ‘Databases’ radio button and your screen will change to this:

Which is fine …but also a little disappointing, because the first thing you’ll notice is that Cloud Control knows nothing about any databases you may be running! But that’s to be expected, since you haven’t gone through the ‘adding targets’ phase of using Cloud Control! That comes next…

10.0 Adding Hosts as Targets

To make Cloud Control useful, you need to add ‘targets’ to it -things it can monitor and report on. Let’s assume that I’ve built an Alpher/Bethe RAC before getting to this point and that both are running their RAC database happily, in the background: in that case, there are two hosts to tell Cloud Control about. There is also a single cluster and a single database to tell it about. We’ll take that step-by-step… and in this step, we’re going to start by telling it about the hosts.

You start by clicking the giant ‘cog wheel’ button at the top-right part of the screen, select Add Target -> Add Targets Manually:

Doing so changes the screen displayed:

You now want to install the Cloud Control agent (=’the monitoring software’) onto the Alpher and Bethe hosts, so click the [Install Agent on Host] button. There’s quite a lot going on in the screen which then appears:

Basically, I’m listing each of the hosts I’ve previously built as part of the Churchill framework into the middle of this screen (there’s no need to add Churchill itself). I’ve told the screen that all hosts are running on the same platform, and I’ve specified that platform as ‘Linux x86_64’. When you’re ready, you click [Next].

The main thing to do on this screen is to state whereabouts on Alpher and Bethe the Cloud Control agent software should be installed. I’ve chosen /u01/app/oracle/product/ccagent13, though the specific directory name at the end of that path is arbitrary. The setting for ‘instance directory’ is generated automatically once you’ve typed something in the ‘installation base’ field. The other thing to do here is: wipe out the contents of the ‘Privileged Delegation Setting’. The field, as shown here, should be entirely empty. This means you’ll have to run a script as root by hand, once the installation process concludes.

Finally, click the big ‘+’ button at the end of the ‘Named Credential’ and ‘Root Credential’ entries. Fill in the named credential as though you were logging on to Alpher and Bethe as the user ‘oracle’ (so, the password is oracle, too):

The same sort of thing applies for the root credential:

Notice at this point that the ‘Mandatory Inputs’ item in the middle-right of the screen has changed from having a red exclamation mark on it to having a green tick: you can click [Next] when this happens. A summary screen is then displayed which you can review at your leisure. But click the [Deploy Agent] button to make the Cloud Control software actually get pushed to the Alpher and Bethe hosts in turn.

It will take a while for the installation to be successful.

You can keep clicking the ‘Refresh’ button manually, or just wait for the screen to auto-refresh every 30 seconds (by default). Either way, you’ll see different progress indicators displayed for the various operations being performed on the various hosts. You should see this sort of thing after about a minute, for example:

…showing that at least the pre-requisite checks have completed successfully on both Alpher and Bethe.

Give it another couple of minutes or so, though, and you’ll end up with this:

…which shows that agents have been deployed successfully to both components of my RAC. Now click the Targets button at the top of the screen and then the Hosts option within that:

Here you’ll see proof that Cloud Control knows about Alpher and Bethe (and, incidentally, about its own host: Wilson). It knows that all the hosts are up and running and that there are no incidents or bad things happening on any of them. You can click on any of the listed host-names to delve deeper into what is running on that specific host: for example, let’s say I click on the link to alpher:

Information is a bit sparse at this point still, because I’ve only just deployed the agent and I haven’t asked Cloud Control to discover what else might be running on the host targets yet. But at least you can see that Wilson and its Cloud Control is starting to be a single-point of management for all the other servers in my Churchill framework.

11.0 Adding Cluster Targets

Host targets are fine -but the fun starts when you get Cloud Control to discover that there are databases and ASM instances running on the discovered host targets. So lets add them into the mix. Start by clicking on the cog-wheel icon at the top of the page once more, then selecting Add Target -> Add Targets Manually. From the screen that appears, this time click on the central option: to Add Using Guided Process:

Select the ‘Oracle Cluster and High Availability Service’ item, as shown, then click [Add]. A rather blank screen then appears:

Click on that little blue ‘magnifying glass’ icon, next to the Specify Host window:

The hosts which you’d already discovered are listed. You just need to highlight one of them (I’ve gone for Alpher) and click the [Select] button. That returns you to the earlier screen -but this time, the name of a host to investigate will be shown in the Specify Host window. Now you just click the [Discover Target] button:

It won’t take long for Cloud Control to work out that Alpher’s part of a cluster called ‘racscan’ -and that Bethe is part of that cluster too. At this point, just click [Save] to add the newly-discovered cluster as a target for Cloud Control to work with, and then click [Close] to finish the job.

12.0 Adding Database Targets

After discovering hosts and clusters, you can complete this initial exploration of Cloud Control by discovering the databases and ASM instances which are running on the freshly-found clusters. So, back to the Target Discovery page (i.e., click the settings cog-wheel, then Add Target -> Add Targets Manually). Click the middle option once more: use the guided process. This time, select the option to discover Oracle databases, listeners and ASM instances:

The screen changes and asks you to specify a host or cluster. Click the magnifying glass icon next to the host/cluster name field:

This time, select the racscan target and click [Select]. Back on the main page, click [Next]. After a short period, the results of the discovery process will be displayed:

It has successfully discovered that there’s a database called “orcl” and an ASM instance and (lower down on the screen and not shown in that screenshot) that there’s a multitude of listeners to manage. It is asking you to supply the password for the dbsnmp and asmsnmp users in the Monitor Password fields you can see above.

You may not remember setting a password for asmsnmp, but you did: it’s part of the Grid Infrastructure install. See slide 7 in the GI installation slideshow of Section 6 on this page, for example. You also specified a password for dbsnmp -see slide 22 of Section 8 of this page, for example. Hopefully you now remember what you supplied at that point! Don’t worry if you don’t, though, because I’ll show you how to set them now anyway.

For dbsnmp, you just connect to either Alpher or Bethe as oracle, log onto the database and unlock the account and set a new password like so:

ssh oracle@192.168.8.101
sqlplus / as sysdba
alter user dbsnmp identified by oracle account unlock;

(Use of “oracle” as a password is not exactly best security practice! I use it here just for convenience at home).

For asmsnmp, you do something very similar, except that you must log on to Alpher or Bethe as grid, not oracle. The commands would be:

ssh grid@192.168.8.101
sqlplus / as sysasm
alter user asmsnmp identified by oracle account unlock;

Now you definitely know the right password in each case, type it in the ‘monitor password’ field and click [Next].

This summary page shows you what Cloud Control has discovered… all you need do is click [Save] to make the discoveries permanent. Click [Close] when prompted.

Now you can click the Targets icon on the top part of the screen, then the Databases item:

It’s still not the most exciting screen in the world, but at least it lists a database now! Click on the link to ‘orcl’ and you’ll see something similar to this:

Now, at last, we see useful information about what the cluster is doing and how well (or badly) it’s coping. You can begin to click on links to specific SQL statements that are being run and investigate their performance and how they might be improved.

If you wanted to just look at what was running on only Alpher or only Bethe, you can click on the link to Instances, select one from the pop-up that then appears, and the screen would then list just that instance’s work instead:

There is obviously a huge amount of clicking around to be done at this point: an article of this sort cannot hope to be anything but a mere taster of what Cloud Control is capable of. Entire books exist on the subject, after all! The point of this article, however, was simply to demonstrate how the Churchill framework helps you build a server that can at least run Cloud Control successfully, and how you can use Cloud Control’s discovery tools to make the Churchill framework’s component servers part of a centrally-managed database infrastructure.

13.0 Conclusion

As I’ve already hinted, Cloud Control is a monster of a subject and this article is not meant as any sort of real guide to its capabilities. But in this article, I’ve shown you how to build a server that is capable of running Cloud Control using the fact that Churchill exists and is running in the background. With essentially one quick command (“sk=5”) specified at boot-up time, all the correct bits and pieces are seamlessly stitched together to create a new server called Wilson that can run Cloud Control successfully.

I’ve shown you how to install Oracle 12c database software on it; how to then run DBCA to create a database suitable for acting as a Cloud Control repository; and how to install Cloud Control itself. I’ve then shown you how to use Cloud Control’s target discovery tools to discover, in turn, hosts, clusters and databases (in that order).

I finished off by giving you a glimpse of the sort of power Cloud Control gives to the DBA, to be able to see what exactly is running in any part of the database, clustered or otherwise, at any given moment.

 

One word of warning, though: Once you’ve administered a database with Cloud Control, it’s very hard to give up the convenience it provides! When a user complains that their SQL is running slowly, being able to click on the precise SQL ID to see, instantly, a graph of what it’s been waiting on:

…and then to be able to click to tune the query or to stabilise its execution plan… well, there’s no real command-line equivalent for any of that! Not that you couldn’t issue a mountain of SQL in SQL*Plus to achieve the same thing, of course… but that’s hardly as convenient!

So it’s fine to learn Cloud Control: future employers will certainly appreciate your finely-honed Cloud Control skills… but they may not be able to afford you actually practising them!

Don’t, in short, get so carried away with the wonders of Cloud Control that you forget how to manage your databases using nothing but a keyboard and a terminal session!