MySQL Replication with SSL Error_code: 2026 – Resolved

We keep returning to an issue with MySQL Replication whereby we keep seeing error code 2026 when trying to set it up with SSL:
[ERROR] Slave I/O: error connecting to master ‘———@———-:—-‘ – retry-time: 60 retries: 86400, Error_code: 2026

The simplest answer is – DONT USE OPENSSL. Mysql comes bundled with yassl, just specify –with-ssl on the configure line rather than adding your openssl path, and mysql will compile with the internal yassl. Same config works just fine with yassl, and doesnt with openssl.

Are there security implications with doing this? Possibly as yassl has had issues in the past. But if you’re opening up your MySQL servers to the public then you’re probably already weighing up risk vs functionality anyway.

Just for the record, some key points from the my.cnf files:
[client]
timezone = UTC
ssl-ca=/etc/certs/ca-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem

[server]
ssl
ssl-ca=/etc/certs/ca-cert.pem
ssl-cert=/etc/certs/server-cert.pem
ssl-key=/etc/certs/server-key.pem

And the MASTER config:
| Slave_IO_State | Master_Host | Master_User | Master_Port | Connect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File | Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Running | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignore_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | Last_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Condition | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File | Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seconds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Error | Last_SQL_Errno | Last_SQL_Error |
| Waiting for master to send event | IPADDRESSS | mysqlrepl | PORT | 60 | mysql_bin_log.000539 | 1652774 | relay-bin.000006 | 1652923 | mysql_bin_log.000539 | Yes | Yes | | | | | | | 0 | | 0 | 1652774 | 1750495 | None | | 0 | Yes | /etc/certs/ca-cert.pem | /etc/certs/ | /etc/certs/client-cert.pem | | /etc/certs/client-key.pem | 0 | No | 0 | | 0 | |

Hopefully that helps someone else get things set up….

Shared Hosting Support for PHP 5.3

Just letting everybody know that as of tonight, we now have servers running in most of our POPs that are capable of running PHP 5.3. If it’s possible on your server, you will see the option in the RackCorp portal to select:

PHP 5.3.6 (SSL,FTP,ZLIB,IMAP,IMAPSSL,GD,MySQL,Sockets,Kerberos,ICONV,MBSTRING,MBREGEX,GetText,MimeMagic)

It takes about 1-2 minutes for your site to do a changeover between versions. If it doesn’t come up, select 5.2 to put things back to how they were, and drop a ticket in to see if we can debug it for you.

We’ve also taken this upgrade to try and get our servers all running the same version of our web manager. By doing so, a lot of you should now find that you can control several more php.ini options for your accounts, and that you also have access to PHP 4.4.9. If you’re unusure of any options, just run with the defaults as they’re pretty safe.

Overall, just remember that PHP 5.3 is not entirely compatible with PHP 5.2. Check out this link for sure:

http://www.php.net/manual/en/migration53.deprecated.php

Some things we came across just in the RackCorp portal:
Depreciated functions:

# ereg() (use preg_match() instead)
# ereg_replace() (use preg_replace() instead)
# eregi() (use preg_match() with the ‘i’ modifier instead)
# eregi_replace() (use preg_replace() with the ‘i’ modifier instead)
# split() (use preg_split() instead)
# spliti() (use preg_split() with the ‘i’ modifier instead)
# session_register() (use the $_SESSION superglobal instead)
# session_unregister() (use the $_SESSION superglobal instead)
# session_is_registered() (use the $_SESSION superglobal instead)
# set_socket_blocking() (use stream_set_blocking() instead)
# mysql_db_query() (use mysql_select_db() and mysql_query() instead)
# mysql_escape_string() (use mysql_real_escape_string() instead)

Also, for the few of you out there that use it (like us!) Ming has been moved out to PECL.

Otherwise, we’ll let you know how we go with our migration over the next few months. We’re particularly interested in any SOAP changes as things seem to silently have crept in during PHP 5.2.X revisions that although minor changes left us with big headaches (timezones in datetime fields – ARGH!)… To be continued…. 🙂

FPM Migration from PHP 5.2 to PHP 5.3

Some installation/configuration notes we came up with whle integrating PHP 5.3 into our standard deployment system:

Compile options
PHP 5.3 has FPM inbuilt now, so there’s no longer any need to apply diffs to the source, but you DO still have to specify “–enable-fpm”.  All other FPM parameters have been depreciated now (except the run as user/group ones – but we use chrootuid anyway).  That’s all that’s required for compilation.

Configuration
Things have changed significantly here, from an XML format into an ini format.  Here’s a typical fpm.ini that we’re using:

[global]
pid = /var/log/php-fpm.pid
error_log = /var/log/php-fpm.log
log_level = notice

emergency_restart_threshold = 10
emergency_restart_interval = 1m
process_control_timeout = 5s
daemonize = yes

[www]
;this is the IP/port to listen for fastcgi requests on
listen = 127.0.0.1:41493
listen.backlog = 1024
listen.allowed_clients = 127.0.0.1
; not sure if we need to specify user/group here, but it’s indicated it is required, but if we chrootuid php so it’s already running as another use, it seems to be ignored
user = nobody
group = nobody

; This stuff actually works in PHP 5.3 – and works well!!
pm = dynamic
pm.max_children = 3
pm.start_servers = 1
pm.min_spare_servers = 1
pm.max_spare_servers = 1
pm.max_requests = 500

request_terminate_timeout = 40s
request_slowlog_timeout = 300s
slowlog = /var/log/php-fpm.log.slow
rlimit_files = 2048

rlimit_core = 0
catch_workers_output = no

Runtime
Things have changed here slightly too, we used to run:

bin/php-cgi –fpm

and php would load up the compile-time default configuration file.

With 5.3, please note that the binary to run has changed and moved – it’s in the sbin directory and is called php-fpm. We also have to pass in the fpm configuration ini file as a parameter (as there wasn’t a compile time option for this anymore):

sbin/php-fpm –fpm-config=/conf/php-fpm.ini

And that’s it. We’ve also tested out the dynamic process spawning as this was of most interest to us as on shared servers RAM is like gold! Seems to work well – it seems to guage the necessity for additional processes by whether it has to wait for a request to be served. It seems to drop off unused processes after 10 seconds or so – couldn’t find anything about this so I guess it’s just some magical internal algorithm, but from my tests it looks to work and work well.

SSL Certificate 2048 bit generation with OpenSSL

openssl req -nodes -newkey rsa:2048 -keyout www.examplewebsite.com.key -out www.examplewebsite.com.csr

You'll be prompted for a series of questions:
Country Name (2 letter code) [GB]:AU
State or Province Name (full name) [Berkshire]:New South Wales
Locality Name (eg, city) [Newbury]:Sydney
Organization Name (eg, company) [My Company Ltd]:Example Company
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:www.examplewebsite.com
Email Address []:

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

You then upload the contents of the www.examplewebsite.com.csr file to your SSL certificate registrar and they'll send you back your certificate file.  You can then plug these into your favorite webserver/application server.

sysctl.conf and other settings for high performance webservers

There’s a couple of key settings on CentOS servers that significantly helps for high performance web servers that we always put in by default across all of our managed machines:

• net.ipv4.tcp_syncookies = 1
  While it’s more commonly seen by people wanting to prevent denial of service attacks from taking down their websites, some people don’t realise that a heavy traffic site is not much different from one that is under a constant denial of service!

• net.ipv4.netfilter.ip_conntrack_max = 300000
  Netfilter under linux does a great job, but it can sometimes be artificially restricted by some OS limitations that try to prevent some traffic from taking up too many system resources.  This is one of those settings that I feel is often set too low.  We ramp it up to 300,000 which means NetFilter can track up to 300,000 “sessions” (such as a HTTP connection) at one time.  If you’ve got 10,000 people on your website at once, you’ll definitely want to adjust this one!

• net.ipv4.tcp_max_syn_backlog = 10240
  An application such as Nginx is very capable of serving as many TCP connections as an operating system and hardware can handle.  With that said, there will be a backlog of TCP connections ins a pending state before the user-space application such as Nginx gets to call accept().  The key here is to make sure the backlog of unaccepted TCP connections never exceeds the above number else there will be the equivalent of packetloss of the connection packets, and some clients will experience delays, if not a complete outage.  We find 10240 is a high enough number for this on current modern day servers.

• net.core.netdev_max_backlog = 4000
  This one is important, particularly for servers that operate past 100MBit/s.  It governs how many packets will be queued inbetween the kernel processing the interface packet queue.  At gigabit speeds on busy servers, seeing the queue exceed the default of 1000 is pretty common.  We usually put this up to 4,000 for web servers.

• kernel.panic = 10
  While unrelated to performance, there’s nothing worse on a busy web server than seeing a kernel panic.  While this isn’t common, when you do push a server to it’s limits, you can certainly come across kernel panics more commonly than you might otherwise, and this setting just helps reduce downtime on production servers.

We usually also change the TCP congestion control algorithm too by adding the following to rc.local:

• /sbin/modprobe tcp_htcp

You will also want to increase the send queue on your interface by adding the following to your rc.local (you’ll want to change eth0 to your interface name):

• /sbin/ifconfig eth0 txqueuelen 10000

There’s a lot of commentary online about changing tcp memory buffers and sizes.  Personally I haven’t found them to make much difference on a suitably spec’d server.  One day I might get around to having a look at how these affect performance, but for now, the above settings are known to achieve gigabit HTTP serving speeds for our webservers so that’s good enough for me!