Category: Research & Dev

Virtual Server Image Backups – Performance Improvement

Tonight we’ve deployed the latest version of our Virtual Server backup manager. This software is responsible for:
1) Creating LVM snapshots of Virtual Servers
2) Reading entire snapshots, 1MB at a time and generating data hashes
3) Comparing those data hashes to our backup repositories (usually in off-site datacenters)
4) Copying any changed blocks to our backup repositories

Traditionally, most of our customers run on small VMs (20GB-30GB), so the above process has worked quite well albeit not optimal. This year we’ve seen a significant uptake of large 200GB+ VM’s which has made us re-evaluate our backup manager. Tonight’s update is a significant one in terms of performance:

  • Change 20150516-001: Backup Repository no longer re-calculates hashes during a backup. They’re calculated once when the backup is first written, used for comparison purposes, and updated as new blocks come in.
  • Change 20150601-001: snapshot blkio limiting. Previously, backup processes were allowed to utilise full sequential read speeds due to oversight in the blkio settings. This has now been remedied, with sequential reads now limited to 60% of the virtual server’s allocated device speed.
  • Change 20150601-002: posix_fadvise(POSIX_FADV_SEQUENTIAL) utilisation. Added libc binding to make a call to posix_fadvise to set the POSIX_FADV_SEQUENTIAL flag. This should mean the Linux Disk Cache doesn’t get blown away during backups.

Coming up in the future will be the much requested feature addition to allow file-level mounts of backups!

[] [Digg] [StumbleUpon] [Technorati] [Windows Live]

Brocade VDX Switches – line protocol down / ISL DOWN

It would seem that it’s rather difficult (not possible?) to deactivate the ISL exchange between Brocade switches. This normally isn’t a problem unless you really want to segment your Brocade switches by not running ISL trunking between certain ones. As soon as you want to do this you may come across your interface in this state:

line protocol down (ISL DOWN)

Points you in the direction of ISL but when you’ve explicitly disabled fabric ISL for the interface:

no fabric isl
no fabric trunk

You may be left wondering how to disable it. We tried a few things but the Brocade ISL seems to operate at a lower level than the commands presented in NOS. Anyway the eventual solution was simply to change the vcs ID:

vcs vcsid XXX

Where XXX is different to your other switches.

It’s probably the cleaner solution in any case. Had we set things up properly in the lab in the first place then we wouldn’t have encountered this issue at all, but then again we now have the knowledge that NOS is doing things behind the scenes at a lower level than we’ve really got access to (when vcs mode is enabled). I guess it’s always nice to discover this in a lab environment first anyway.

Our first production deployment of Brocade VDX6740 & VDX6710 switches will be occurring later this week in Perth Australia. We’re really enjoying the multi-chassis LACP within a vcs fabric feature (I’m not sure any other switching fabrics support this?). It gives us an opportunity to do some fancy redundancy deployments – at least that’s how it’s working in the lab – watch this space for some real world experience coming up soon!

[] [Digg] [StumbleUpon] [Technorati] [Windows Live]

CentOS convert hda to vda for virtualisation / grub problems & root panic

When virtualising a server, it sometimes become necessary to convert an old centos version from using hda as it’s root device, to be referenced by vda for virtio.  Here’s a howto for people who already know what they’re kinda doing:

First thing is to make sure you’re running on the latest kernel for your OS before doing anything to make sure you’ve got the appropriate drivers:
yum update kernel

Step to is to edit some files on there. If you’ve already got it booted then great, otherwise you can boot it using ide virtualisation options in your virtualisation software, or you can use a boot/live cd (lots of options here! If you’re stuck, just use a centos LIVE CD)

Once in there, you’ll need to mount your image’s partitions in order to get into your image’s /boot/grub (or equivalent thereof – if your lost here then this howto really isn’t for you sorry!)

edit and change
(hd0)     /dev/hda
(hd0)     /dev/vda

Next step is to edit your /etc/modprobe.conf and add/update the corresponding virtio modules:

alias eth0 virtio_net
alias scsi_hostadapter1 virtio_blk

(Remember if you’re using a boot cd, make sure you’re editing your image’s files – not your live cd’s temp files!)

Then you need to recreate the initrd file to have the virtio drivers (no redhat won’t put them in by default if they weren’t already there)

mkinitrd -f –with=virtio_blk –with=virtio_pci –builtin=xenblk /boot/initrd-2.6.18-custom.img 2.6.18-308.16.1.el5

Where 2.6.18-308.16.1.el5 is your kernel version (you can get it by looking at your files in /boot
Now you have the choice of altering your /boot/grub/menu.lst to use this new initrd-2.6.18-custom.img file (or you can just reboot, hit a key at grub boot time, use the e key to edit the initrd line – you can use tab to auto-complete to make it easy).

You might also need to update the root= part to be your /dev/vda3 (or whatever your root drive is). Mine was using labels so it didnt need to be changed.

Once booted, you now want to reinstall the stock centos initrd / kernel (it’s always scary to run on your own customisations!)… you’ll want to run:
rpm –erase –nodeps kernel-2.6.18-308.16.1.el5
(Or whatever your kernel version is)
yum install kernel

then reboot and if everything went according to plan, you should be sitting in a stock standard centos kernel, with your drives running on virtio / /dev/vda. Woo!

[] [Digg] [StumbleUpon] [Technorati] [Windows Live]

BIOS Firmware Update ISO for SuperMicro Boards

The use of floppy images for updating Bios images on SuperMicro boards such as the X9SCL or X9SCM are long over.  The images are far too big (8MB).  So the only way to do the bios update is to build a bootable DOS ISO, built with the firmware files.

This is easier said than done ofcourse, but here’s some simple instructions if you’re using windows that you can follow:


Go here:
Download a tool called
Unzip it on your local computer (don’t follow these instructions without unziping)
64-Bit OS users will have additional step to ALSO go to and download Unzip it then copy the mkisofs.exe and cygwin1.dll files from the zip file and put it into the FDOEMCD.builder folder from above (replacing any existing version of these files).

Step 2: BIOS Flash Files

Go here:  (Or your equivalent for your BIOS), and download the correct version for your motherboard – warning – getting it wrong can permanentely destroy your motherboard – warning – 🙂
Unzip the contents of your bios download into the CDROOT folder.  You should have files such as AFUDOSU.SMC and AMI.bat.  If not then this tutorial isn’t for you (sorry!)

Now because the ISO will be read only, we need to shortcut the AMI.bat file a little bit….
create a new batch file called ami2.bat and put the following in it:
afudosu.exe %1 /FDT /MER /OPR

Now start up a dos prompt in windows and go to the corresponding FDOEMCD.builder folder and run makeiso.bat.

You now have fdoem.iso.  This is your bootable ISO with all your files on it (yay!)

Mount it using your favorite IPMI or burn it to CD and use it.  We just mount it using Supermicro’s IPMI.

Once it has booted, just run “ami2.bat x9scm2.608” (or equivalent) and away you go.


A big thanks to the guys that made FDOEMCD Builder.  Without that tool this BIOS update ISO building process would be a lot harder!


POST-INSTALL NOTE: Bios 2.00 which is on the SuperMicro website at this time seems to have a bug on some dual ethernet motherboards.  (such as the X9SCL!).  SuperMicro knows about it and has an app that you can add to the ISO as per the above instructions which can fix the ethernet port.  I don’t want to publicly make available a link to SuperMicro’s patching software but if you’re really desparate like we were then let us know….

[] [Digg] [StumbleUpon] [Technorati] [Windows Live]

Shared Hosting Support for PHP 5.3

Just letting everybody know that as of tonight, we now have servers running in most of our POPs that are capable of running PHP 5.3. If it’s possible on your server, you will see the option in the RackCorp portal to select:


It takes about 1-2 minutes for your site to do a changeover between versions. If it doesn’t come up, select 5.2 to put things back to how they were, and drop a ticket in to see if we can debug it for you.

We’ve also taken this upgrade to try and get our servers all running the same version of our web manager. By doing so, a lot of you should now find that you can control several more php.ini options for your accounts, and that you also have access to PHP 4.4.9. If you’re unusure of any options, just run with the defaults as they’re pretty safe.

Overall, just remember that PHP 5.3 is not entirely compatible with PHP 5.2. Check out this link for sure:

Some things we came across just in the RackCorp portal:
Depreciated functions:

# ereg() (use preg_match() instead)
# ereg_replace() (use preg_replace() instead)
# eregi() (use preg_match() with the ‘i’ modifier instead)
# eregi_replace() (use preg_replace() with the ‘i’ modifier instead)
# split() (use preg_split() instead)
# spliti() (use preg_split() with the ‘i’ modifier instead)
# session_register() (use the $_SESSION superglobal instead)
# session_unregister() (use the $_SESSION superglobal instead)
# session_is_registered() (use the $_SESSION superglobal instead)
# set_socket_blocking() (use stream_set_blocking() instead)
# mysql_db_query() (use mysql_select_db() and mysql_query() instead)
# mysql_escape_string() (use mysql_real_escape_string() instead)

Also, for the few of you out there that use it (like us!) Ming has been moved out to PECL.

Otherwise, we’ll let you know how we go with our migration over the next few months. We’re particularly interested in any SOAP changes as things seem to silently have crept in during PHP 5.2.X revisions that although minor changes left us with big headaches (timezones in datetime fields – ARGH!)… To be continued…. 🙂

[] [Digg] [StumbleUpon] [Technorati] [Windows Live]