Multiple A DNS records and browser retries failover / redundancy

I wanted to know how firefox / chrome etc behave when there are two A DNS records for a website, and one of those IPs does not respond.  The use-case for this is to have the same website hosted in two locations, and when one server/network becomes offline that the other one can take over.

I couldn’t find any up to date info about this so I thought I’d just do some tests and compile my own info. Test case used only two A records for the same hostname.

NOTE: If the connection returns an error sooner, presumably these browsers will failover faster. The test here was a complete drop of traffic with no TCP reset/icmp error coming back from the first attempted IP.

Using multiple DNS A records is a plausible failover mechanism in modern browsers!

PROBLEM: Browsers seem to switch between IPs after a period of inactivity (even just a few minutes).  If your site requires state, you will need to make state be stored inside the browser (preferably) or some kind of shared state system between servers.

Pros:
Multi-server failover is relatively easy, and fast in modern browsers
Traffic is balanced between both locations

Cons:
If one A record server stops responding, the user may experience a delay when loading the site
If one A record server stops responding while the user was already on the site, they may see an error until they reload the page.
If state required, need to store in browser or shared state system

Virtual Server Image Backups – Performance Improvement

Tonight we’ve deployed the latest version of our Virtual Server backup manager. This software is responsible for:
1) Creating LVM snapshots of Virtual Servers
2) Reading entire snapshots, 1MB at a time and generating data hashes
3) Comparing those data hashes to our backup repositories (usually in off-site datacenters)
4) Copying any changed blocks to our backup repositories

Traditionally, most of our customers run on small VMs (20GB-30GB), so the above process has worked quite well albeit not optimal. This year we’ve seen a significant uptake of large 200GB+ VM’s which has made us re-evaluate our backup manager. Tonight’s update is a significant one in terms of performance:

• Change 20150516-001: Backup Repository no longer re-calculates hashes during a backup. They’re calculated once when the backup is first written, used for comparison purposes, and updated as new blocks come in.
• Change 20150601-001: snapshot blkio limiting. Previously, backup processes were allowed to utilise full sequential read speeds due to oversight in the blkio settings. This has now been remedied, with sequential reads now limited to 60% of the virtual server’s allocated device speed.
• Change 20150601-002: posix_fadvise(POSIX_FADV_SEQUENTIAL) utilisation. Added libc binding to make a call to posix_fadvise to set the POSIX_FADV_SEQUENTIAL flag. This should mean the Linux Disk Cache doesn’t get blown away during backups.

Coming up in the future will be the much requested feature addition to allow file-level mounts of backups!

Brocade VDX Switches – line protocol down / ISL DOWN

It would seem that it’s rather difficult (not possible?) to deactivate the ISL exchange between Brocade switches. This normally isn’t a problem unless you really want to segment your Brocade switches by not running ISL trunking between certain ones. As soon as you want to do this you may come across your interface in this state:

line protocol down (ISL DOWN)

Points you in the direction of ISL but when you’ve explicitly disabled fabric ISL for the interface:

no fabric isl
no fabric trunk

You may be left wondering how to disable it. We tried a few things but the Brocade ISL seems to operate at a lower level than the commands presented in NOS. Anyway the eventual solution was simply to change the vcs ID:

vcs vcsid XXX

Where XXX is different to your other switches.

It’s probably the cleaner solution in any case. Had we set things up properly in the lab in the first place then we wouldn’t have encountered this issue at all, but then again we now have the knowledge that NOS is doing things behind the scenes at a lower level than we’ve really got access to (when vcs mode is enabled). I guess it’s always nice to discover this in a lab environment first anyway.

Our first production deployment of Brocade VDX6740 & VDX6710 switches will be occurring later this week in Perth Australia. We’re really enjoying the multi-chassis LACP within a vcs fabric feature (I’m not sure any other switching fabrics support this?). It gives us an opportunity to do some fancy redundancy deployments – at least that’s how it’s working in the lab – watch this space for some real world experience coming up soon!

CentOS convert hda to vda for virtualisation / grub problems & root panic

When virtualising a server, it sometimes become necessary to convert an old centos version from using hda as it’s root device, to be referenced by vda for virtio.  Here’s a howto for people who already know what they’re kinda doing:

First thing is to make sure you’re running on the latest kernel for your OS before doing anything to make sure you’ve got the appropriate drivers:
yum update kernel

Step to is to edit some files on there. If you’ve already got it booted then great, otherwise you can boot it using ide virtualisation options in your virtualisation software, or you can use a boot/live cd (lots of options here! If you’re stuck, just use a centos LIVE CD)

Once in there, you’ll need to mount your image’s partitions in order to get into your image’s /boot/grub (or equivalent thereof – if your lost here then this howto really isn’t for you sorry!)

edit device.map and change
(hd0)     /dev/hda
to
(hd0)     /dev/vda

Next step is to edit your /etc/modprobe.conf and add/update the corresponding virtio modules:

alias eth0 virtio_net
alias scsi_hostadapter1 virtio_blk

(Remember if you’re using a boot cd, make sure you’re editing your image’s files – not your live cd’s temp files!)

Then you need to recreate the initrd file to have the virtio drivers (no redhat won’t put them in by default if they weren’t already there)

mkinitrd -f –with=virtio_blk –with=virtio_pci –builtin=xenblk /boot/initrd-2.6.18-custom.img 2.6.18-308.16.1.el5

Where 2.6.18-308.16.1.el5 is your kernel version (you can get it by looking at your files in /boot
Now you have the choice of altering your /boot/grub/menu.lst to use this new initrd-2.6.18-custom.img file (or you can just reboot, hit a key at grub boot time, use the e key to edit the initrd line – you can use tab to auto-complete to make it easy).

You might also need to update the root= part to be your /dev/vda3 (or whatever your root drive is). Mine was using labels so it didnt need to be changed.

Once booted, you now want to reinstall the stock centos initrd / kernel (it’s always scary to run on your own customisations!)…..so you’ll want to run:
rpm –erase –nodeps kernel-2.6.18-308.16.1.el5
(Or whatever your kernel version is)
yum install kernel

then reboot and if everything went according to plan, you should be sitting in a stock standard centos kernel, with your drives running on virtio / /dev/vda. Woo!

BIOS Firmware Update ISO for SuperMicro Boards

The use of floppy images for updating Bios images on SuperMicro boards such as the X9SCL or X9SCM are long over.  The images are far too big (8MB).  So the only way to do the bios update is to build a bootable DOS ISO, built with the firmware files.

This is easier said than done ofcourse, but here’s some simple instructions if you’re using windows that you can follow:

Step 1: FDOEMCD

Go here: http://www.fdos.org/bootdisks/
Download a tool called FDOEMCD.builder.zip
Unzip it on your local computer (don’t follow these instructions without unziping)
64-Bit OS users will have additional step to ALSO go to http://smithii.com/cdrtools/ and download cdrtools-latest.zip. Unzip it then copy the mkisofs.exe and cygwin1.dll files from the zip file and put it into the FDOEMCD.builder folder from above (replacing any existing version of these files).

Step 2: BIOS Flash Files

Go here: http://www.supermicro.com/support/bios/  (Or your equivalent for your BIOS), and download the correct version for your motherboard – warning – getting it wrong can permanentely destroy your motherboard – warning – 🙂
Unzip the contents of your bios download into the CDROOT folder.  You should have files such as AFUDOSU.SMC and AMI.bat.  If not then this tutorial isn’t for you (sorry!)

Now because the ISO will be read only, we need to shortcut the AMI.bat file a little bit….
rename AFUDOSU.SMC to AFUDOSU.EXE
create a new batch file called ami2.bat and put the following in it:
afudosu.exe %1 /FDT /MER /OPR

Now start up a dos prompt in windows and go to the corresponding FDOEMCD.builder folder and run makeiso.bat.

You now have fdoem.iso.  This is your bootable ISO with all your files on it (yay!)

Mount it using your favorite IPMI or burn it to CD and use it.  We just mount it using Supermicro’s IPMI.

Once it has booted, just run “ami2.bat x9scm2.608” (or equivalent) and away you go.

A big thanks to the guys that made FDOEMCD Builder.  Without that tool this BIOS update ISO building process would be a lot harder!

POST-INSTALL NOTE: Bios 2.00 which is on the SuperMicro website at this time seems to have a bug on some dual ethernet motherboards.  (such as the X9SCL!).  SuperMicro knows about it and has an app that you can add to the ISO as per the above instructions which can fix the ethernet port.  I don’t want to publicly make available a link to SuperMicro’s patching software but if you’re really desparate like we were then let us know….