Dizwell Informatics

News from Nowhere

Perform periodic zpool scrubs

Scrubbing is what we call the process of checking the integrity of data stored on your ZFS volumes. The basic command is very simple: zpool scrub name-of-zpool. Issue that (as root) and ZFS will go through every block of data on your zpool and compare its checksum with the stored checksum: if the two checksums agree, the data is fine. If they don’t, then ZFS will repair the data until they do, assuming your zpool uses some form of data redundancy (such as raidz), which implies another copy of the data is available from another disk.

In short, scrubbing is very important and very useful… and it doesn’t happen automatically! This is presumably because all that data reading and checksumming hammers the hell out of your disks and CPU. However, scrubs are performed with low priority -sufficiently low that you’re not supposed to ‘feel’ it happening.

So, it’s costly, but more than extremely useful, and you have to make it happen by invoking a script in the dead hours of the early morning.

Here’s the script I use:

# Script expects to be able to record its progress in a /root/logs directory.

export PATH=/usr/sbin:/usr/bin
LOG=/root/logs/`date +20\%y\%m\%d\%H\%M`-scrubs.log
zpool list -H -o name | while read pool; do

# If a scrub is already running, then don't try and start a second one (because you can't!)
 if zpool status -v $pool | grep -q "scrub in progress"; then
 echo "Can't scrub pool "$pool" -scrub already running!" >> $LOG 2>&1
 echo "Scrubbing pool "$pool >> $LOG 2>&1
 zpool scrub $pool

echo "Script took $SECONDS seconds to complete (scrubs run for hours in background!)." >> $LOG 2>&1
exit 0

It basically iterates around a listing of every pool you’ve got. For each pool, it detects whether a scrub is already running: you can only have one scrub per pool at a time, so attempting to start a second simultaneously would just produce an error. If it detects an existing scrub is in operation for a pool, the script records the fact in a log file, and moves on to the next zpool it knows about. If no existing scrub is running, it invokes the zpool scrub command for that pool.

Note that the zpool scrub command returns almost instantly, no matter how big your zpool is because it’s run asynchronously in the background. That bit of “script took $SECONDS to complete” is there not to measure the length of time that the scrub takes to complete, therefore, but simply to record that the invoking script itself ran successfully to completion.

Once you’ve saved your equivalent script, don’t forget to make it executable:

chmod 775 scrubs.sh

My logs directory (which I have to create before the script will run, but I do so as a standard feature of setting up any new O/S, so I haven’t bothered putting in code to automatically create it if it doesn’t already exist, easy enough to do though that would be) then starts to look like this:

root@bach:~/logs# ls -l
total 6
-rw-r--r--   1 root     root         245 Apr 25 12:18 201604251209-copyfromrvw.log
-rw-r--r--   1 root     root          80 Apr 25 16:56 201604251656-scrubs.log
-rw-r--r--   1 root     root         144 Apr 25 17:06 201604251706-scrubs.log

…and you can see I’ve got a couple of test-runs of the scrubs.sh script happening. Look at the second of those logs and its contents will be as follows:

root@bach:~/logs# cat 201604251706-scrubs.log 
Scrubbing pool rpool
Can't scrub pool safedata -scrub already running!
Script took 1 seconds to complete (scrubs run for hours in background!).

You can see that the rpool pool was scrubbed successfully, but before an attempt was made to scrub my safedata pool, the script noted that a previous scrub was already underway, so didn’t try starting a second one. (Incidentally, if you wanted your script to stop a pre-existing scrub and start a new one, you could do that too: zpool srub -s pool-name is the command to stop a running scrub.)

Having proved the script works, I add the following lines to my root crontab (with the crontab -e command):

# Perform a scrub of ZFS volume at 2:01AM on the First of each month
1 2 1 * * /root/scripts/scrubs.sh 2>&1

That means I will now be performing a monthly scrub of my all-important safedata zpool, which is probably sufficiently frequent to keep me happy -though my servers only store data which is shipped to them from a PC that also runs a ZFS mirror… and that mirror gets scrubbed weekly. If I didn’t have that independent check of data integrity going on, I don’t think a monthly scrub would be frequent enough myself.

You need to check the status of your zpools manually to see how any running scrubs are progressing and if the scrubs have found any errors. For example:

root@bach:~/logs# zpool status rpool
  pool: rpool
 state: ONLINE
  scan: scrub repaired 0 in 38s with 0 errors on Mon Apr 25 17:07:03 2016


        rpool     ONLINE       0     0     0
          c1t1d0  ONLINE       0     0     0

errors: No known data errors

That’s how you know a scrub has finished, that it took 38 seconds to complete (my rpool is pretty small -around 120GB on a solid state hard disk) and it didn’t detect any problems.

And here’s what an on-going scrub looks like:

root@bach:~/logs# zpool status safedata
  pool: safedata
 state: ONLINE
  scan: scrub in progress since Mon Apr 25 16:56:31 2016
    734G scanned out of 7.32T at 360M/s, 5h20m to go
    0 repaired, 9.79% done

        NAME        STATE     READ WRITE CKSUM
        safedata    ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            c3t0d0  ONLINE       0     0     0
            c3t1d0  ONLINE       0     0     0
            c3t2d0  ONLINE       0     0     0
            c3t3d0  ONLINE       0     0     0

errors: No known data errors

So this one has processed 734GB, but hasn’t finished yet (because this pool is 7.3TB in size and is made up from 4 bits of slow, spinning rust). Hence the ‘scrub in progress’ message (which is the text my shell script uses to detect pre-running scrubs, of course). There have been no data errors detected so far (which is nice), but I basically need to come back in 5 hours and 20 minutes’ time to find out if the entire pool was detected ‘clean’.

So, to sum up: scrubbing is something you must do regularly. Anything less than monthly is probably not frequent enough. Use cron to schedule your scrubs in the dead of night. And remember to run the zpool status command occasionally to find out whether your scrubs detected and/or fixed any data corruption issues.