Another in the series of the fun things that are new about CentOS 7: setting up an iSCSI target. Asquith used to do this with this sort of code:

lvcreate -l 70%VG -n asquith-dbdata vg1
echo "<target iqn.2013-08.home.dizwell:asquith.dbdata>" >> /etc/tgt/targets.conf
echo "       backing-store /dev/vg1/asquith-dbdata" >> /etc/tgt/targets.conf
echo "</target>" >> /etc/tgt/targets.conf
chkconfig tgtd on
service tgtd start

This is all good, old-fashioned stuff involving writing something to a configuration file and then starting a daemon to use it.

But this is not how we do it now. Oh no. In CentOS 7 (and its Red Hat-y and Oracle-y equivalents, of course), we use an interactive tool called targetcli to invoke an independent ‘shell’ in which these sorts of commands are issued instead:

backstores/block create name=dbdata dev=/dev/mapper/vg1-asquith--dbdata
iscsi/ create iqn.2014-07.home.dizwell:asquith.dbdata
cd iscsi/iqn.2014-07.home.dizwell:asquith.dbdata/tpg1
portals/ create
luns/ create /backstores/block/dbdata
set attribute authentication=0
set attribute generate_node_acls=1

…and the final exit there takes you back to your original shell. (I probably should say that targetcli itself is not actually that new, having first been released in about 2009… but it’s now the default way of doing things in the 7 release of Enterprise Linux, and that’s new -at least, to me!).

Anyway, targetcli is definitely nice and easy and there are no services to worry about: it all just starts getting shared by magic. About the only way to check anything is actually working after you’ve issued all those targetcli commands is to do:

netstat -ant

…before and after. If port 3260 is not in use before, but is in use afterwards, then you know it’s working properly.

The real bummer about the new technique, however, is that it’s not really very scriptable. It’s an interactive tool after all, and appears to expect a system admin to be sitting at the end of the keyboard… which is not much use if you’re trying to get this all to happen as part of a Kickstart auto-build, say!

I did work out that the old trick of piping things together will help. For example, if the above commands are re-written slightly to be:

echo "cd /" | targetcli
echo "backstores/block create name=dbdata dev=/dev/mapper/vg1-asquith--dbdata" | targetcli
echo "iscsi/ create iqn.2014-07.home.dizwell:asquith.dbdata" | targetcli
echo "cd iscsi/iqn.2014-07.home.dizwell:asquith.dbdata/tpg1" | targetcli
echo "portals/ create" | targetcli
echo "luns/ create /backstores/block/dbdata" | targetcli
echo "set attribute authentication=0" | targetcli
echo "set attribute generate_node_acls=1" | targetcli

…then each of the commands in double-quotes will be passed through to the targetcli shell in turn and executed in just the same way as if you’d typed things interactively.

Excellent… but I need these commands in a slightly different context. What I’m after is for Kickstart to create a shell script that contains these commands so that it can then execute that shell script later on to automagically set up iSCSI target sharing when building a CentOS 7 Asquith Server. That means I need Kickstart to run commands which create a script which contains these commands. And at that point, I’m asking for the commands to be re-written in the following manner:

echo "echo \"cd /\" | targetcli" >> /root/
echo "echo \"backstores/block create name=dbdata dev=/dev/mapper/vg1-asquith--dbdata\" | targetcli" >> /root/
echo "echo \"iscsi/ create iqn.2014-07.home.dizwell:asquith.dbdata\" | targetcli" >> /root/
echo "echo \"cd iscsi/iqn.2014-07.home.dizwell:asquith.dbdata/tpg1\" | targetcli" >> /root/
echo "echo \"portals/ create\" | targetcli" >> /root/
echo "echo \"luns/ create /backstores/block/dbdata\" | targetcli" >> /root/
echo "echo \"set attribute authentication=0\" | targetcli" >> /root/
echo "echo \"set attribute generate_node_acls=1\" | targetcli" >> /root/

The levels of indirection here start to do my head in!

Take the first command: echo “echo \”cd /\“ | targetcli” » /root/

That’s my earlier echo-and-pipe-to-targetcli command wrapped up in an echo command of its own. Why? Because Kickstart will perform the ‘outer echo’ and thus write a line of text reading just echo “cd /” | targetcli to a shell script called So when Kickstart later runs, the correct targetcli command is finally run.

So basically, we’re nesting an echo inside an echo. Kickstart will run the ‘outer echo’ so that the ‘inner echo’ command gets written to a script file. Only when that script is itself later run will the ‘inner echo’ actually be run and do anything.

Now, the way I’ve written my original inner echoes is perhaps peculiar to me: double quotes make visual sense to me and there’s no major difference in Bash between scripting with double or single quotes. Without a major functional difference, I prefer doubles. But if you start nesting your echoes, the “inner echo” has to escape its double quotes, otherwise they get taken literally, as characters, not command delimiters.

In other words, the command we eventually want to run might be:

echo "cd /" | targetcli

…where the double-quotes are not escaped, because they delimit what is to be echoed. But the command we have to run to get this command written into a shell script is:

echo "echo \"cd /\" | targetcli" >> /root/

So the ‘outer’ echo is now a command to write the words echo “<something>“ into a shell script -but the double-quotes used by this inner echo have to be regarded as literal text, not as parts of the outer echo command. Hence they need to be preceded by a “\” to turn them into literals (“escaped”, in the lingo).

Is your head hurting yet?! It gets worse (a bit)!

Remember that these echoed echo commands are being written into a shell script. Shell scripts need to start with a line saying where the shell executables are to be found. In the world of the Bourne Again Shell, that means starting things with a line which reads:


Now, we want a command that echoes that into a shell script, before we later go on to execute the shell script. No problems …we just do this:

echo "#!/bin/bash" > /root/

This is merely as before: we’re wrapping the command we eventually want executed inside an echo statement, using double quotes as delimiters to define what gets echoed, and finishing off with a redirection out to the shell script that will contain the command.

Except that it won’t work:

[[email protected] ~]# echo "#!/bin/bash" > /root/
-bash: !/bin/bash": event not found

You might think that one or more of the “shebang” characters (the ”#!” at the start of the line being echoed) need escaping, so that something like

echo "\#\!/bin/bash" > /root/

…would do the trick. And indeed, the above escaped command will “work” instead of producing an ‘event not found’ error, but it doesn’t work very well! Just try displaying the contents of the file created with that modified command:

[[email protected] ~]# cat

This shows you that the escape characters have actually become part of the contents of the shell script, rather than interpreted as escape characters. Present as literals, though, the escape characters mean the shell script can’t actually work when invoked. So this won’t do.

Odd though it might seem at first sight, this behaviour is actually perfectly cromulant and well-documented in the Bash manual. Specifically:

A double quote may be quoted within double quotes by preceding it with a backslash. If enabled, history expansion will be performed unless an ‘!’ appearing in double quotes is escaped using a backslash. The backslash preceding the ‘!’ is not removed.

So what’s the fix? Well, the simplest I can think of is to …er, use single quotes. You’ll find that:

echo '#!/bin/bash' > /root/

…works in the sense of not itself returning an error AND works in the sense that it writes the correct command into the shell script file we’re trying to create.

But now, at this point, you realise it’s a bit silly to have a mix of single and double-quotes in the same set of commands, so you think that you could go back to the original targetcli commands and re-write them using single quotes (after all, the manual makes it clear that there’s no real difference between single and double quotes except for the way four literal characters are treated).

This means your ‘doing it directly’ commands would be written as:

echo 'cd /' | targetcli
echo 'backstores/block create name=dbdata dev=/dev/mapper/vg1-asquith--dbdata' | targetcli
echo 'iscsi/ create iqn.2014-07.home.dizwell:asquith.dbdata' | targetcli
echo 'cd iscsi/iqn.2014-07.home.dizwell:asquith.dbdata/tpg1' | targetcli
echo 'portals/ create' | targetcli
echo 'luns/ create /backstores/block/dbdata' | targetcli
echo 'set attribute authentication=0' | targetcli
echo 'set attribute generate_node_acls=1' | targetcli

And they do, indeed, all work as advertised in this form. But now you want to apply that ‘layer of indirection’ that comes from the fact that you’re writing a script to write a script… so you might end up with this:

echo '#!/bin/bash' > /root/
echo 'echo 'cd /' | targetcli' >> /root/
echo 'echo 'backstores/block create name=dbdata dev=/dev/mapper/vg1-asquith--dbdata' | targetcli' >> /root/
echo 'echo 'iscsi/ create iqn.2014-07.home.dizwell:asquith.dbdata' | targetcli' >> /root/
echo 'echo 'cd iscsi/iqn.2014-07.home.dizwell:asquith.dbdata/tpg1' | targetcli' >> /root/
echo 'echo 'portals/ create' | targetcli' >> /root/
echo 'echo 'luns/ create /backstores/block/dbdata' | targetcli' >> /root/
echo 'echo 'set attribute authentication=0' | targetcli' >> /root/
echo 'echo 'set attribute generate_node_acls=1' | targetcli' >> /root/

I have to say, I don’t like the look of that, because gut instinct tells me double-quotes are needed there somewhere (and since, in this case, single and double quotes are interchangeable, there’s no reason why you couldn’t come up with a rule that says ‘inner echoes get doubles, outer echoes use singles’ and thus end up with something that looks a bit clearer than the above and yet remains consistent!).

But it will all work.

Which is the main thing 8-o

Though my head still hurts!

(Incidentally, you don’t need to point knowingly, laughing your head off all the while, exclaiming that I should have used a here document technique to avoid all those nested echoes in the first place. It’s true and I know (and knew) it. But lots of single-line echoes are how I start writing stuff. Only when I know it works do I start applying layers of ‘elegance’ by code tidy-ups such as that. This blog was dedicated to those whose heads hurt, not those who can write shell scripts more elegantly than me… there are far too many of those!)

CentOS 7 and Kickstart

I have long since given up hoping that things which work fine in one version of Red Hat/CentOS/etc will continue to work in the next version, unmolested. But it’s still darn’d annoying when stuff you know works fine in version X breaks in slightly mysterious ways in version X+1. Kickstart (the tool for automating CentOS/Red Hat deployments) is a case in point.

A Kickstart file which worked fine for CentOS 5.x, for example, turns out to contain entirely the wrong syntax as far as CentOS 6.x is concerned -so a re-write is required to make it functional once more.

The bad news is that this pattern continues with CentOS 7, to the point where even invoking a Kickstart installation has changed (and accordingly cost me quite a lot of time tracking down precisely where the problem is).

Short version: in the past, you invoked a Kickstart installation by pressing Tab on the first boot menu and then typing something like ks=hd:sr1/kickstart.ks. Now you do it by pressing Tab on the first boot menu and typing ks=cdrom:/dev/sr1:/kickstart.ks. It’s a subtle change but it makes all the difference!

Longer version: the change is explained pretty well in this bug report for Fedora 19 (Red Hat 7 and its clones is really a re-jigged version of Fedora 18/19, so the bug reports for the one often apply to the other).

Actually, the new syntax is clearer and more logical than the old, so I probably shouldn’t complain… but I’m in that sort of the mood at the moment, so I will :-)

There are lots of other changes, too, of course: software package groups have changed, which makes a version 6.x script useless for doing 7.x installs, just for starters.

All of which is by way of explanation for late delivery on promised RH7-ish versions of Asquith and Salisbury. They are coming (update: no they’re not!), but somewhat more slowly than I had anticipated, because of the “fun” I’ve been having with version 7’s Kickstart!

It’s been a bit of a bad week, really. First my eye op.

Second, the production SQL Server box went on the fritz and a dozen or more of us, including specialists from the States, spent a couple of days throwing everything we could at it without effect. In thirty-odd years, I have never seen a problem that couldn’t be resolved by some analysis and problem-solving… but this one couldn’t. We worked so late on Thursday trying to fix things that I ended up checked in to a city hotel rather than attempt the 2 hour journey home… only to then have to spend all Saturday night and the wee small hours of this morning working from home as we moved what had been a sensitive database running on a fairly feeble virtual machine onto a freshly-commissioned physical beast of a box. The problems appear to have been mostly resolved as a result, but I am still currently on line trying to get one or two loose ends tidied up.

And third, to cap it all off and make a bad week really miserable, my 18 year-old cat, Lucretia, who has been looking ever-more feeble for the past six months or so really looked terrible this morning and so we decided we finally had no choice but to pay a visit to the vet and have her put to sleep. He did it beautifully and we are left feeling we did exactly the right thing at about the right time. But for the first time in 18 years, my home is completely cat-less. There are still the wallabies, but they don’t quite count as someone who could sit on my shoulder and purr terrifically in my ear.

And now I must get back to trying to fix SQL Server to Oracle replication on our new production box.

Sometimes, life’s a bitch.

Post Op, ergo propter hoc

It wasn’t fun, exactly, but it wasn’t open heart surgery either and my eye now only feels slightly as if it had a run-in with a large bouncer from a seedy nightclub. Three lots of drops four times a day are a nuisance, too (I keep missing and my face ends up rather wetter than it ought), but apparently infection is the big nightmare and I’ll do anything to keep it at bay if I can.

So, other than on-going maintenance issues, I am now the proud posssessor of a bionic left eye which seems to perform quite well. It will take a month or so until everything settles down enough to make getting a new pair of glasses worthwhile, so in the meantime I’m using a -4.0 glasses lens with an eye that has now been set to -2.0, courtesy of the implant… and everything looks slightly blurry in consequence. But close-up to computer monitors and the like, where I have been in the habit of removing glasses for years, it’s now actually my right eye that is doing it tough, for it was a -2.75 and so is slightly worse than my new one.

It turns out that it has a nascent cataract in it too, though, so next year I intend to do the same to it, and have it replaced with a lens to match my new left one. I’d do it sooner if I could, I think, except that it costs a fiendish amount of money that I don’t have. So I’ve a year of saving to do, and some finger-crossing to hope that the right eye doesn’t turn into a galloping cataract like the left one did in the meantime.

For being such a brave little soldier, I get to stay at home for three days (I think I’ll be back to work tomorrow) and am under strict instructions not to do any housework for at least a couple of months. There is a silver lining to be found in all things, then :-)

CentOS 7 arrives…

CentOS 7 has been released and is available for download from the usual places. I’ll be adapting (or trying to!) Salisbury and Asquith to work with it over the next few days and you can expect updates to those frameworks as I do.

I go for my cataract removal operation next Monday, however, so although I will at home for a couple of days afterwards (and thus have plenty of time to do the deed), I might still be bumping into things and thus not at my most efficient. Asquith/Salisbury v.2 when I can, therefore, but no promises.