Nginx and PHP-FPM for heavy load wordpress web server with high traffic 2000+ concurrent connections.

To make it to read short I will not be describing all the time I lost during last half year trying to find out the “right settings” for the high traffic WordPress web server running I was responsible for, and this was on the top of my primary job, and that’s why it took so much time of blind testing and playing parameters with live production server, and as you understand I have no ability to make sharp actions and have long downtime, while having only 30-40 minutes per day. Anyway I found all needed information, tested everything, it works, I’m happy.

So the server is quite ordinary 2 CPU (8 cores total), 16 GB RAM, SSD drive, colocated in proper data center. Ubuntu OS, PHP 5.3.5 (with PHP5-FPM), MySQL 5.5, WordPress 3.3.1, APC cache.

Just to give you some ideas what I was trying while I was loosing my time. As far as you understand I couldn’t reinstall OS, so I was playing only with the packages installed on it. So as for database I use MySQL 5.1 after 5.1a and at the end 5.5. Honestly I didn’t find any big advantages that made my life easier or brought some performance benefits only simply because of a new version.

As for PHP, the first version I’ve started with already running on this server was PHP 5.1.6, after it was 5.2.3, then 5.2.5, then 5.2.17, then 5.3.3, and at last 5.3.5. Why I’m enumerating all these versions ? Early versions of PHP do not include the PHP5-FPM package, and means it was not comfortable to apply special patch contained this package. Version PHP 5.3.8 on latest Ubuntu server has perfectly runnig PHP5-FPM out of the box.

Now about PHP cache. I’ve tried e-accelerator at the beginning, then it was xcache and at last I’ve grounded on APC. I cannot answer why APC, but I simply never had any problem with, that is what I can not tell about two others, especially about xcache.

So now you are aware about these processes that preceded my successful completion of optimization and possibly your tried something similar already to do, so now I will describe it more in details.

So first of all and most important is OS limitations, this is where I’ve lost most of my time, trying to implement somebodies advices without any success.

Here is my Linux Kernel configuration file, /etc/sysctl.conf

fs.file-max = 262144

kernel.pid_max = 262144

net.ipv4.tcp_rmem = 4096 87380 8388608

net.ipv4.tcp_wmem = 4096 87380 8388608

net.ipv4.netfilter.ip_conntrack_max = 65536

net.core.rmem_max = 25165824

net.core.rmem_default = 25165824

net.core.wmem_max = 25165824

net.core.wmem_default = 131072

net.core.netdev_max_backlog = 8192

net.ipv4.tcp_window_scaling = 1

net.core.optmem_max = 25165824

net.core.somaxconn = 65536

net.ipv4.ip_local_port_range = 1024 65535

kernel.shmmax = 4294967296

vm.max_map_count = 262144

You can easily find in the man of all the descriptions of these parameters, but what I wish to mention that this configuration is good for my 16GB RAM server. I suppose more RAM means bigger values possible.

But all these settings above are useless without fixing ulimit parameter for you Linux. For Debian based Linux it should be done in /etc/security/limits.conf

it should be updated for any user which responsible for your running package

nginx soft nofile 131072

nginx soft nofile 131072

mysql soft nofile 131072

mysql soft nofile 131072

root soft nofile 131072

root soft nofile 131072

and don’t forget to add in /etc/pam.d/common-session the following string:

session required pam_limits.so

So here you are! Your server can breathe in much freely!

Now the database.

First and the main this is to switch your MySQL into InnoDB. Yes, it take more memory than MyISAM, it harder to process defragmentation for them, but the performance is why you are here on this page, so InnoDB and no more discussions.

Before I will publish my configuration file for MySQL, there is another important step you should do. Create a RAM disk of 1-2 GB for MySQL “temp” purposes (various caches, tables, and others).

For Debian based Linux it should be done in /etc/fstab:

ramdisk /tmp tmpfs mode=1777,size=2048m

To check that it’s working use command df -h

you will get something like this:

ramdisk 2.0G 140M 1.9G 7% /tmp

Now the /etc/mysql/my.cnf file (tuned in suffers, tears and blood)

[mysqld]

user = mysql

socket = /var/run/mysqld/mysqld.sock

port = 3306

basedir = /usr

datadir = /var/lib/mysql

tmpdir = /tmp

default-storage-engine = InnoDB

skip-external-locking

bind-address = 127.0.0.1

key_buffer = 512K

max_allowed_packet = 16M

thread_stack = 128K

thread_cache_size = 48

thread_concurrency = 8

innodb_file_io_threads = 8

myisam-recover = BACKUP

max_connections = 768

table_cache = 2048

query_cache_type = 1

query_cache_limit = 32M

query_cache_size = 384M

query_cache_min_res_unit = 256

query_prealloc_size = 65K

range_alloc_block_size = 128K

read_rnd_buffer_size = 1M

record_buffer = 1M

read_buffer_size = 4M

join_buffer_size = 2M

sort_buffer_size = 2M

bulk_insert_buffer_size = 8M

long_query_time = 2

sync_binlog = 1

log_bin = /var/log/mysql/mysql-bin.log

expire_logs_days = 5

max_binlog_size = 100M

innodb_additional_mem_pool_size = 64M

innodb_buffer_pool_size = 4096M

innodb_log_file_size = 256M

innodb_log_files_in_group = 2

innodb_log_buffer_size = 32M

innodb_open_files = 16384

open-files-limit = 16384

max_tmp_tables = 1024

table_open_cache = 2048

#tmp_table_size = 512M

#max_heap_table_size = 512M

wait_timeout = 1200

As I said, explanations will be later on, all parameters could be found in on MySQL manual.

Now most annoying, but where important for overall productivity of PHP Web server – PHP5-FPM.

I will provide configuration of 3 files:

php.ini

main.conf

www.conf

I change only 4 parameters in this file /etc/php5/fpm/php.ini:

memory_limit = 256M

post_max_size = 8M

upload_max_filesize = 10M

max_file_uploads = 20

and add support of APC with the following string:

extension=/usr/lib/php5/20090626/apc.so

This is it for php.ini

Now php handler parameters which could be found in /etc/php5/fpm/main.conf:

[global]

pid = /var/run/php5-fpm.pid

error_log = /var/log/php5-fpm.log

emergency_restart_threshold = 0

emergency_restart_interval = 0

process_control_timeout = 0

daemonize = yes

include =/etc/php5/fpm/pool.d/*.conf

And now /etc/php5/fpm/pool.d/www.conf:

[www]

listen = 127.0.0.1:9000

;listen = /tmp/php5-fpm.sock

listen.backlog = 65536

user = www-data

group = www-data

pm = dynamic

pm.max_children = 16

;pm.start_servers = 12

;pm.min_spare_servers = 8

;pm.max_spare_servers = 16

pm.max_requests = 1536

pm.status_path = /status

request_terminate_timeout = 0

rlimit_files = 65536

rlimit_core = unlimited

catch_workers_output = no

Please not that all configuration files above are not for simple copy paste but for understanding the idea why the values are like these.

It’s not recommended to use PHP5-FPM via unix socket for more than 300+ concurrent connections due to you will definitely get nasty error: sock failed (11: Resource temporarily unavailable) or connect() failed (11: Resource temporarily unavailable) depending on your version, (you may need to increase start_servers, or min/max_spare_servers), and and every 5^th page will be return Error 502. By using tcp/ip as a handler you will get more stability but will loose around 10 connection on each 100 concurrent connection in performance.

And the other 2 tricky things to make it to run more smoothly is to gracefully restart PHP5-FPM every 9 minutes and clear the Query Cache on MySQL every 8 minutes. Again, the time depends on your load and content of your project.

For doing that you need Cron up and running and two files with special commands to execute:

To set the cron, you need crontab -e to set the time of execute and path to the files:

*/8 * * * * /etc/cron.my/mysql_query_flash

*/9 * * * * /etc/init.d/php5-fpm reload

Here the script of the /etc/cron.my/mysql_query_flash file:

mysql -u root -pYOURPASSWORD -e “flush query cache”;

Nginx configuration. Nothing interesting and tricky here.

As for/etc/nginx/nginx.conf file here you are:

user www-data;

worker_processes 8;

pid /var/run/nginx.pid;

timer_resolution 100ms;

worker_rlimit_nofile 32768;

worker_priority -5;

events {

worker_connections 4096;

multi_accept on;

use epoll;

}

http {

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 60;

types_hash_max_size 2048;

server_tokens off;

server_names_hash_bucket_size 64;

server_name_in_redirect off;

client_body_buffer_size 8K;

client_header_buffer_size 2k;

client_max_body_size 16k;

large_client_header_buffers 4 2k;

include /etc/nginx/mime.types;

default_type application/octet-stream;

access_log off ; #/var/log/nginx/access.log;

error_log /var/log/nginx/error.log;

gzip on;

gzip_disable “msie6”;

gzip_vary on;

gzip_proxied any;

gzip_comp_level 6;

gzip_buffers 64 8k;

gzip_min_length 1024; gzip_http_version 1.1;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

include /etc/nginx/conf.d/*.conf;

}

APC configuration /etc/php5/conf.d/apc.ini I left in default setting but only increased the RAM parameter:

apc.shm_size = 128M

For now this is it. I use STATUS module from nginx that is quite informative and here is the average result during the Saturday and Sunday when I have the highest load.

I’m quite proud of it!

Active connections: 2617
server accepts handled requests
 62950 62950 300560
Reading: 198 Writing: 81 Waiting: 2338

Since PHP 7 all these settings are less important. PHP-FPM is now rock solid solution for any project.