.ssh authorized_keys and centos / redhat / selinux enabled Linux solution

Just a quick note that if you ever create /home or /home/user or /home/user/.ssh or /home/user/.ssh/authorized_keys that you’ll need to reset the selinux contexts for the coresponding files/directories or else sshd won’t be allowed to access your authorized_keys file!

If your /home isn’t very large (and /home was what you created), the easiest way of fixing things is to run this:

restorecon -R -v /home

Otherwise if you added /home via useradd AND the user home dir then you can probably get away with just:

restorecon -R -v /home/user

This will reset the contexts placed on /home/user/.ssh/*

BIOS Firmware Update ISO for SuperMicro Boards

The use of floppy images for updating Bios images on SuperMicro boards such as the X9SCL or X9SCM are long over.  The images are far too big (8MB).  So the only way to do the bios update is to build a bootable DOS ISO, built with the firmware files.

This is easier said than done ofcourse, but here’s some simple instructions if you’re using windows that you can follow:

Step 1: FDOEMCD

Go here: http://www.fdos.org/bootdisks/
Download a tool called FDOEMCD.builder.zip
Unzip it on your local computer (don’t follow these instructions without unziping)
64-Bit OS users will have additional step to ALSO go to http://smithii.com/cdrtools/ and download cdrtools-latest.zip. Unzip it then copy the mkisofs.exe and cygwin1.dll files from the zip file and put it into the FDOEMCD.builder folder from above (replacing any existing version of these files).

Step 2: BIOS Flash Files

Go here: http://www.supermicro.com/support/bios/  (Or your equivalent for your BIOS), and download the correct version for your motherboard – warning – getting it wrong can permanentely destroy your motherboard – warning – 🙂
Unzip the contents of your bios download into the CDROOT folder.  You should have files such as AFUDOSU.SMC and AMI.bat.  If not then this tutorial isn’t for you (sorry!)

Now because the ISO will be read only, we need to shortcut the AMI.bat file a little bit….
rename AFUDOSU.SMC to AFUDOSU.EXE
create a new batch file called ami2.bat and put the following in it:
afudosu.exe %1 /FDT /MER /OPR

Now start up a dos prompt in windows and go to the corresponding FDOEMCD.builder folder and run makeiso.bat.

You now have fdoem.iso.  This is your bootable ISO with all your files on it (yay!)

Mount it using your favorite IPMI or burn it to CD and use it.  We just mount it using Supermicro’s IPMI.

Once it has booted, just run “ami2.bat x9scm2.608” (or equivalent) and away you go.

A big thanks to the guys that made FDOEMCD Builder.  Without that tool this BIOS update ISO building process would be a lot harder!

POST-INSTALL NOTE: Bios 2.00 which is on the SuperMicro website at this time seems to have a bug on some dual ethernet motherboards.  (such as the X9SCL!).  SuperMicro knows about it and has an app that you can add to the ISO as per the above instructions which can fix the ethernet port.  I don’t want to publicly make available a link to SuperMicro’s patching software but if you’re really desparate like we were then let us know….

MySQL Replication with SSL Error_code: 2026 – Resolved

We keep returning to an issue with MySQL Replication whereby we keep seeing error code 2026 when trying to set it up with SSL:
[ERROR] Slave I/O: error connecting to master ‘———@———-:—-‘ – retry-time: 60 retries: 86400, Error_code: 2026

The simplest answer is – DONT USE OPENSSL. Mysql comes bundled with yassl, just specify –with-ssl on the configure line rather than adding your openssl path, and mysql will compile with the internal yassl. Same config works just fine with yassl, and doesnt with openssl.

Are there security implications with doing this? Possibly as yassl has had issues in the past. But if you’re opening up your MySQL servers to the public then you’re probably already weighing up risk vs functionality anyway.

Just for the record, some key points from the my.cnf files:
[client]
timezone = UTC
ssl-ca=/etc/certs/ca-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem

[server]
ssl
ssl-ca=/etc/certs/ca-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem

And the MASTER config:
| Slave_IO_State | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |
| Waiting for master to send event | IPADDRESSS | mysqlrepl | PORT | 60 | mysql_bin_log.000539 | 1652774 | relay-bin.000006 | 1652923 | mysql_bin_log.000539 | Yes | Yes | | | | | | | 0 | | 0 | 1652774 | 1750495 | None | | 0 | Yes | /etc/certs/ca-cert.pem | /etc/certs/ | /etc/certs/client-cert.pem | | /etc/certs/client-key.pem | 0 | No | 0 | | 0 | |

Hopefully that helps someone else get things set up….

Shared Hosting Support for PHP 5.3

Just letting everybody know that as of tonight, we now have servers running in most of our POPs that are capable of running PHP 5.3. If it’s possible on your server, you will see the option in the RackCorp portal to select:

PHP 5.3.6 (SSL,FTP,ZLIB,IMAP,IMAPSSL,GD,MySQL,Sockets,Kerberos,ICONV,MBSTRING,MBREGEX,GetText,MimeMagic)

It takes about 1-2 minutes for your site to do a changeover between versions. If it doesn’t come up, select 5.2 to put things back to how they were, and drop a ticket in to see if we can debug it for you.

We’ve also taken this upgrade to try and get our servers all running the same version of our web manager. By doing so, a lot of you should now find that you can control several more php.ini options for your accounts, and that you also have access to PHP 4.4.9. If you’re unusure of any options, just run with the defaults as they’re pretty safe.

Overall, just remember that PHP 5.3 is not entirely compatible with PHP 5.2. Check out this link for sure:

http://www.php.net/manual/en/migration53.deprecated.php

Some things we came across just in the RackCorp portal:
Depreciated functions:

# ereg() (use preg_match() instead)
# ereg_replace() (use preg_replace() instead)
# eregi() (use preg_match() with the ‘i’ modifier instead)
# eregi_replace() (use preg_replace() with the ‘i’ modifier instead)
# split() (use preg_split() instead)
# spliti() (use preg_split() with the ‘i’ modifier instead)
# session_register() (use the $_SESSION superglobal instead)
# session_unregister() (use the $_SESSION superglobal instead)
# session_is_registered() (use the $_SESSION superglobal instead)
# set_socket_blocking() (use stream_set_blocking() instead)
# mysql_db_query() (use mysql_select_db() and mysql_query() instead)
# mysql_escape_string() (use mysql_real_escape_string() instead)

Also, for the few of you out there that use it (like us!) Ming has been moved out to PECL.

Otherwise, we’ll let you know how we go with our migration over the next few months. We’re particularly interested in any SOAP changes as things seem to silently have crept in during PHP 5.2.X revisions that although minor changes left us with big headaches (timezones in datetime fields – ARGH!)… To be continued…. 🙂

FPM Migration from PHP 5.2 to PHP 5.3

Some installation/configuration notes we came up with whle integrating PHP 5.3 into our standard deployment system:

Compile options
PHP 5.3 has FPM inbuilt now, so there’s no longer any need to apply diffs to the source, but you DO still have to specify “–enable-fpm”.  All other FPM parameters have been depreciated now (except the run as user/group ones – but we use chrootuid anyway).  That’s all that’s required for compilation.

Configuration
Things have changed significantly here, from an XML format into an ini format.  Here’s a typical fpm.ini that we’re using:

[global]
pid = /var/log/php-fpm.pid
error_log = /var/log/php-fpm.log
log_level = notice

emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 5s
daemonize = yes

[www]
;this is the IP/port to listen for fastcgi requests on
listen = 127.0.0.1:41493
listen.backlog = 1024
listen.allowed_clients = 127.0.0.1
; not sure if we need to specify user/group here, but it’s indicated it is required, but if we chrootuid php so it’s already running as another use, it seems to be ignored
user = nobody
group = nobody

; This stuff actually works in PHP 5.3 – and works well!!
pm = dynamic
pm.max_children = 3
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1
pm.max_requests = 500

request_terminate_timeout = 40s
request_slowlog_timeout = 300s
slowlog = /var/log/php-fpm.log.slow
rlimit_files = 2048

rlimit_core = 0
catch_workers_output = no

Runtime
Things have changed here slightly too, we used to run:

bin/php-cgi –fpm

and php would load up the compile-time default configuration file.

With 5.3, please note that the binary to run has changed and moved – it’s in the sbin directory and is called php-fpm. We also have to pass in the fpm configuration ini file as a parameter (as there wasn’t a compile time option for this anymore):

sbin/php-fpm –fpm-config=/conf/php-fpm.ini

And that’s it. We’ve also tested out the dynamic process spawning as this was of most interest to us as on shared servers RAM is like gold! Seems to work well – it seems to guage the necessity for additional processes by whether it has to wait for a request to be served. It seems to drop off unused processes after 10 seconds or so – couldn’t find anything about this so I guess it’s just some magical internal algorithm, but from my tests it looks to work and work well.