Nginx and PHP-FPM for heavy load wordpress web server with high traffic 2000+ concurrent connections.
To make it to read short I will not be describing all the time I lost during last half year trying to find out the “right settings” for the high traffic WordPress web server running I was responsible for, and this was on the top of my primary job, and that’s why it took so much time of blind testing and playing parameters with live production server, and as you understand I have no ability to make sharp actions and have long downtime, while having only 30-40 minutes per day. Anyway I found all needed information, tested everything, it works, I’m happy.
So the server is quite ordinary 2 CPU (8 cores total), 16 GB RAM, SSD drive, colocated in proper data center. Ubuntu OS, PHP 5.3.5 (with PHP5-FPM), MySQL 5.5, WordPress 3.3.1, APC cache.
Just to give you some ideas what I was trying while I was loosing my time. As far as you understand I couldn’t reinstall OS, so I was playing only with the packages installed on it. So as for database I use MySQL 5.1 after 5.1a and at the end 5.5. Honestly I didn’t find any big advantages that made my life easier or brought some performance benefits only simply because of a new version.
As for PHP, the first version I’ve started with already running on this server was PHP 5.1.6, after it was 5.2.3, then 5.2.5, then 5.2.17, then 5.3.3, and at last 5.3.5. Why I’m enumerating all these versions ? Early versions of PHP do not include the PHP5-FPM package, and means it was not comfortable to apply special patch contained this package. Version PHP 5.3.8 on latest Ubuntu server has perfectly runnig PHP5-FPM out of the box.
Now about PHP cache. I’ve tried e-accelerator at the beginning, then it was xcache and at last I’ve grounded on APC. I cannot answer why APC, but I simply never had any problem with, that is what I can not tell about two others, especially about xcache.
So now you are aware about these processes that preceded my successful completion of optimization and possibly your tried something similar already to do, so now I will describe it more in details.
So first of all and most important is OS limitations, this is where I’ve lost most of my time, trying to implement somebodies advices without any success.
Here is my Linux Kernel configuration file, /etc/sysctl.conf
fs.file-max = 262144
kernel.pid_max = 262144
net.ipv4.tcp_rmem = 4096 87380 8388608
net.ipv4.tcp_wmem = 4096 87380 8388608
net.ipv4.netfilter.ip_conntrack_max = 65536
net.core.rmem_max = 25165824
net.core.rmem_default = 25165824
net.core.wmem_max = 25165824
net.core.wmem_default = 131072
net.core.netdev_max_backlog = 8192
net.ipv4.tcp_window_scaling = 1
net.core.optmem_max = 25165824
net.core.somaxconn = 65536
net.ipv4.ip_local_port_range = 1024 65535
kernel.shmmax = 4294967296
vm.max_map_count = 262144
You can easily find in the man of all the descriptions of these parameters, but what I wish to mention that this configuration is good for my 16GB RAM server. I suppose more RAM means bigger values possible.
But all these settings above are useless without fixing ulimit parameter for you Linux. For Debian based Linux it should be done in /etc/security/limits.conf
it should be updated for any user which responsible for your running package
nginx soft nofile 131072
nginx soft nofile 131072
mysql soft nofile 131072
mysql soft nofile 131072
root soft nofile 131072
root soft nofile 131072
and don’t forget to add in /etc/pam.d/common-session the following string:
session required pam_limits.so
So here you are! Your server can breathe in much freely!
Now the database.
First and the main this is to switch your MySQL into InnoDB. Yes, it take more memory than MyISAM, it harder to process defragmentation for them, but the performance is why you are here on this page, so InnoDB and no more discussions.
Before I will publish my configuration file for MySQL, there is another important step you should do. Create a RAM disk of 1-2 GB for MySQL “temp” purposes (various caches, tables, and others).
For Debian based Linux it should be done in /etc/fstab:
ramdisk /tmp tmpfs mode=1777,size=2048m
To check that it’s working use command df -h
you will get something like this:
ramdisk 2.0G 140M 1.9G 7% /tmp
Now the /etc/mysql/my.cnf file (tuned in suffers, tears and blood)
user = mysql
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /var/lib/mysql
tmpdir = /tmp
default-storage-engine = InnoDB
bind-address = 127.0.0.1
key_buffer = 512K
max_allowed_packet = 16M
thread_stack = 128K
thread_cache_size = 48
thread_concurrency = 8
innodb_file_io_threads = 8
myisam-recover = BACKUP
max_connections = 768
table_cache = 2048
query_cache_type = 1
query_cache_limit = 32M
query_cache_size = 384M
query_cache_min_res_unit = 256
query_prealloc_size = 65K
range_alloc_block_size = 128K
read_rnd_buffer_size = 1M
record_buffer = 1M
read_buffer_size = 4M
join_buffer_size = 2M
sort_buffer_size = 2M
bulk_insert_buffer_size = 8M
long_query_time = 2
sync_binlog = 1
log_bin = /var/log/mysql/mysql-bin.log
expire_logs_days = 5
max_binlog_size = 100M
innodb_additional_mem_pool_size = 64M
innodb_buffer_pool_size = 4096M
innodb_log_file_size = 256M
innodb_log_files_in_group = 2
innodb_log_buffer_size = 32M
innodb_open_files = 16384
open-files-limit = 16384
max_tmp_tables = 1024
table_open_cache = 2048
#tmp_table_size = 512M
#max_heap_table_size = 512M
wait_timeout = 1200
As I said, explanations will be later on, all parameters could be found in on MySQL manual.
Now most annoying, but where important for overall productivity of PHP Web server – PHP5-FPM.
I will provide configuration of 3 files:
I change only 4 parameters in this file /etc/php5/fpm/php.ini:
memory_limit = 256M
post_max_size = 8M
upload_max_filesize = 10M
max_file_uploads = 20
and add support of APC with the following string:
This is it for php.ini
Now php handler parameters which could be found in /etc/php5/fpm/main.conf:
pid = /var/run/php5-fpm.pid
error_log = /var/log/php5-fpm.log
emergency_restart_threshold = 0
emergency_restart_interval = 0
process_control_timeout = 0
daemonize = yes
And now /etc/php5/fpm/pool.d/www.conf:
listen = 127.0.0.1:9000
;listen = /tmp/php5-fpm.sock
listen.backlog = 65536
user = www-data
group = www-data
pm = dynamic
pm.max_children = 16
;pm.start_servers = 12
;pm.min_spare_servers = 8
;pm.max_spare_servers = 16
pm.max_requests = 1536
pm.status_path = /status
request_terminate_timeout = 0
rlimit_files = 65536
rlimit_core = unlimited
catch_workers_output = no
Please not that all configuration files above are not for simple copy paste but for understanding the idea why the values are like these.
It’s not recommended to use PHP5-FPM via unix socket for more than 300+ concurrent connections due to you will definitely get nasty error: sock failed (11: Resource temporarily unavailable) or connect() failed (11: Resource temporarily unavailable) depending on your version, (you may need to increase start_servers, or min/max_spare_servers), and and every 5th page will be return Error 502. By using tcp/ip as a handler you will get more stability but will loose around 10 connection on each 100 concurrent connection in performance.
And the other 2 tricky things to make it to run more smoothly is to gracefully restart PHP5-FPM every 9 minutes and clear the Query Cache on MySQL every 8 minutes. Again, the time depends on your load and content of your project.
For doing that you need Cron up and running and two files with special commands to execute:
To set the cron, you need crontab -e to set the time of execute and path to the files:
*/8 * * * * /etc/cron.my/mysql_query_flash
*/9 * * * * /etc/init.d/php5-fpm reload
Here the script of the /etc/cron.my/mysql_query_flash file:
mysql -u root -pYOURPASSWORD -e “flush query cache”;
Nginx configuration. Nothing interesting and tricky here.
As for/etc/nginx/nginx.conf file here you are:
large_client_header_buffers 4 2k;
access_log off ; #/var/log/nginx/access.log;
gzip_buffers 64 8k;
gzip_min_length 1024; gzip_http_version 1.1;
APC configuration /etc/php5/conf.d/apc.ini I left in default setting but only increased the RAM parameter:
apc.shm_size = 128M
For now this is it. I use STATUS module from nginx that is quite informative and here is the average result during the Saturday and Sunday when I have the highest load.
I’m quite proud of it!
Active connections: 2617 server accepts handled requests 62950 62950 300560 Reading: 198 Writing: 81 Waiting: 2338
Since PHP 7 all these settings are less important. PHP-FPM is now rock solid solution for any project.