Nginx and PHP-FPM for heavy load wordpress web server with high traffic 2000+ concurrent connections

Nginx and PHP-FPM for heavy load wordpress web server with high traffic 2000+ concurrent connections.

php

To make it to read short I will not be describing all the time I lost during last half year trying to find out the “right settings” for the high traffic WordPress web server running I was responsible for, and this was on the top of my primary job, and that’s why it took so much time of blind testing and playing parameters with live production server, and as you understand I have no ability to make sharp actions and have long downtime, while having only 30-40 minutes per day. Anyway I found all needed information, tested everything, it works, I’m happy.

So the server is quite ordinary 2 CPU (8 cores total), 16 GB RAM, SSD drive, colocated in proper data center. Ubuntu OS, PHP 5.3.5 (with PHP5-FPM), MySQL 5.5, WordPress 3.3.1, APC cache.

Just to give you some ideas what I was trying while I was loosing my time. As far as you understand I couldn’t reinstall OS, so I was playing only with the packages installed on it. So as for database I use MySQL 5.1 after 5.1a and at the end 5.5. Honestly I didn’t find any big advantages that made my life easier or brought some performance benefits only simply because of a new version.

As for PHP, the first version I’ve started with already running on this server was PHP 5.1.6, after it was 5.2.3, then 5.2.5, then 5.2.17, then 5.3.3, and at last 5.3.5. Why I’m enumerating all these versions ? Early versions of PHP do not include the PHP5-FPM package, and means it was not comfortable to apply special patch contained this package. Version PHP 5.3.8 on latest Ubuntu server has perfectly runnig PHP5-FPM out of the box.

Now about PHP cache. I’ve tried e-accelerator at the beginning, then it was xcache and at last I’ve grounded on APC. I cannot answer why APC, but I simply never had any problem with, that is what I can not tell about two others, especially about xcache.

So now you are aware about these processes that preceded my successful completion of optimization and possibly your tried something similar already to do, so now I will describe it more in details.

So first of all and most important is OS limitations, this is where I’ve lost most of my time, trying to implement somebodies advices without any success.

Here is my Linux Kernel configuration file, /etc/sysctl.conf

fs.file-max = 262144

kernel.pid_max = 262144

net.ipv4.tcp_rmem = 4096 87380 8388608

net.ipv4.tcp_wmem = 4096 87380 8388608

net.ipv4.netfilter.ip_conntrack_max = 65536

net.core.rmem_max = 25165824

net.core.rmem_default = 25165824

net.core.wmem_max = 25165824

net.core.wmem_default = 131072

net.core.netdev_max_backlog = 8192

net.ipv4.tcp_window_scaling = 1

net.core.optmem_max = 25165824

net.core.somaxconn = 65536

net.ipv4.ip_local_port_range = 1024 65535

kernel.shmmax = 4294967296

vm.max_map_count = 262144

You can easily find in the man of all the descriptions of these parameters, but what I wish to mention that this configuration is good for my 16GB RAM server. I suppose more RAM means bigger values possible.

But all these settings above are useless without fixing ulimit parameter for you Linux. For Debian based Linux it should be done in /etc/security/limits.conf

it should be updated for any user which responsible for your running package

nginx soft nofile 131072

nginx soft nofile 131072

mysql soft nofile 131072

mysql soft nofile 131072

root soft nofile 131072

root soft nofile 131072

and don’t forget to add in /etc/pam.d/common-session the following string:

session required pam_limits.so

So here you are! Your server can breathe in much freely!

Now the database.

First and the main this is to switch your MySQL into InnoDB. Yes, it take more memory than MyISAM, it harder to process defragmentation for them, but the performance is why you are here on this page, so InnoDB and no more discussions.

Before I will publish my configuration file for MySQL, there is another important step you should do. Create a RAM disk of 1-2 GB for MySQL “temp” purposes (various caches, tables, and others).

For Debian based Linux it should be done in /etc/fstab:

ramdisk /tmp tmpfs mode=1777,size=2048m

To check that it’s working use command df -h

you will get something like this:

ramdisk 2.0G 140M 1.9G 7% /tmp

Now the /etc/mysql/my.cnf file (tuned in suffers, tears and blood)

[mysqld]

user = mysql

socket = /var/run/mysqld/mysqld.sock

port = 3306

basedir = /usr

datadir = /var/lib/mysql

tmpdir = /tmp

default-storage-engine = InnoDB

skip-external-locking

bind-address = 127.0.0.1

key_buffer = 512K

max_allowed_packet = 16M

thread_stack = 128K

thread_cache_size = 48

thread_concurrency = 8

innodb_file_io_threads = 8

myisam-recover = BACKUP

max_connections = 768

table_cache = 2048

query_cache_type = 1

query_cache_limit = 32M

query_cache_size = 384M

query_cache_min_res_unit = 256

query_prealloc_size = 65K

range_alloc_block_size = 128K

read_rnd_buffer_size = 1M

record_buffer = 1M

read_buffer_size = 4M

join_buffer_size = 2M

sort_buffer_size = 2M

bulk_insert_buffer_size = 8M

long_query_time = 2

sync_binlog = 1

log_bin = /var/log/mysql/mysql-bin.log

expire_logs_days = 5

max_binlog_size = 100M

innodb_additional_mem_pool_size = 64M

innodb_buffer_pool_size = 4096M

innodb_log_file_size = 256M

innodb_log_files_in_group = 2

innodb_log_buffer_size = 32M

innodb_open_files = 16384

open-files-limit = 16384

max_tmp_tables = 1024

table_open_cache = 2048

#tmp_table_size = 512M

#max_heap_table_size = 512M

wait_timeout = 1200

As I said, explanations will be later on, all parameters could be found in on MySQL manual.

Now most annoying, but where important for overall productivity of PHP Web server – PHP5-FPM.

I will provide configuration of 3 files:

php.ini

main.conf

www.conf

I change only 4 parameters in this file /etc/php5/fpm/php.ini:

memory_limit = 256M

post_max_size = 8M

upload_max_filesize = 10M

max_file_uploads = 20

and add support of APC with the following string:

extension=/usr/lib/php5/20090626/apc.so

This is it for php.ini

Now php handler parameters which could be found in /etc/php5/fpm/main.conf:

[global]

pid = /var/run/php5-fpm.pid

error_log = /var/log/php5-fpm.log

emergency_restart_threshold = 0

emergency_restart_interval = 0

process_control_timeout = 0

daemonize = yes

include =/etc/php5/fpm/pool.d/*.conf

And now /etc/php5/fpm/pool.d/www.conf:

[www]

listen = 127.0.0.1:9000

;listen = /tmp/php5-fpm.sock

listen.backlog = 65536

user = www-data

group = www-data

pm = dynamic

pm.max_children = 16

;pm.start_servers = 12

;pm.min_spare_servers = 8

;pm.max_spare_servers = 16

pm.max_requests = 1536

pm.status_path = /status

request_terminate_timeout = 0

rlimit_files = 65536

rlimit_core = unlimited

catch_workers_output = no

Please not that all configuration files above are not for simple copy paste but for understanding the idea why the values are like these.

It’s not recommended to use PHP5-FPM via unix socket for more than 300+ concurrent connections due to you will definitely get nasty error: sock failed (11: Resource temporarily unavailable) or connect() failed (11: Resource temporarily unavailable) depending on your version, (you may need to increase start_servers, or min/max_spare_servers), and and every 5th page will be return Error 502. By using tcp/ip as a handler you will get more stability but will loose around 10 connection on each 100 concurrent connection in performance.

And the other 2 tricky things to make it to run more smoothly is to gracefully restart PHP5-FPM every 9 minutes and clear the Query Cache on MySQL every 8 minutes. Again, the time depends on your load and content of your project.

For doing that you need Cron up and running and two files with special commands to execute:

To set the cron, you need crontab -e to set the time of execute and path to the files:

*/8 * * * * /etc/cron.my/mysql_query_flash

*/9 * * * * /etc/init.d/php5-fpm reload

Here the script of the /etc/cron.my/mysql_query_flash file:

mysql -u root -pYOURPASSWORD -e “flush query cache”;

Nginx configuration. Nothing interesting and tricky here.

As for/etc/nginx/nginx.conf file here you are:

user www-data;

worker_processes 8;

pid /var/run/nginx.pid;

timer_resolution 100ms;

worker_rlimit_nofile 32768;

worker_priority -5;

events {

worker_connections 4096;

multi_accept on;

use epoll;

}

http {

sendfile on;

tcp_nopush on;

tcp_nodelay on;

keepalive_timeout 60;

types_hash_max_size 2048;

server_tokens off;

server_names_hash_bucket_size 64;

server_name_in_redirect off;

client_body_buffer_size 8K;

client_header_buffer_size 2k;

client_max_body_size 16k;

large_client_header_buffers 4 2k;

include /etc/nginx/mime.types;

default_type application/octet-stream;

access_log off ; #/var/log/nginx/access.log;

error_log /var/log/nginx/error.log;

gzip on;

gzip_disable “msie6”;

gzip_vary on;

gzip_proxied any;

gzip_comp_level 6;

gzip_buffers 64 8k;

gzip_min_length 1024; gzip_http_version 1.1;

gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

include /etc/nginx/conf.d/*.conf;

}

APC configuration /etc/php5/conf.d/apc.ini I left in default setting but only increased the RAM parameter:

apc.shm_size = 128M

For now this is it. I use STATUS module from nginx that is quite informative and here is the average result during the Saturday and Sunday when I have the highest load.

I’m quite proud of it!

Active connections: 2617
server accepts handled requests
 62950 62950 300560
Reading: 198 Writing: 81 Waiting: 2338


PHP7

Since PHP 7 all these settings are less important. PHP-FPM is now rock solid solution for any project.

 

14 Comments

  1. i think here is a problem

    nginx soft nofile 131072
    nginx soft nofile 131072
    mysql soft nofile 131072
    mysql soft nofile 131072
    root soft nofile 131072
    root soft nofile 131072

    should be like this?
    nginx soft nofile 131072
    nginx hard nofile 131072

    Reply
  2. Hello Sergio,

    First of all, thank you for your post. I’m using a xen vps with 8 core cpu, 2gb ram & 4gb swap.
    I’m on centos 5.8 32bit with nginxproxy, php53u, php-fpm, xcache.
    Can you tellme what will be my optimal configs to use in this server. I don’t care if the server runs on 99% cpu usage or 1999mb ram. I just want every bit of performance.

    Any help will be helpful.
    Thanks.

    Reply
  3. Not so sure regarding the socks vs TCP. One of our clients has 2000+ concurrent on peaks (3.5-4million pageviews per month), with socks and we have not seen this issue. But we are using nginx as a reverse static cache proxy for Apache/PHP5-FPM/APC backend.

    Reply
  4. Hi, nice article.

    Since you are running nginx as user “www-data”, shouldnt you use
    “www-data soft nofile 131072” intead of “nginx soft nofile 131072” ?

    Actually, it would probably be more explanatory as “web-server-running-username soft nofile 131072”

    Reply
  5. I’m quite curious why you restart PHP-FPM every 9 minutes.

    In fact, I have been considering doing this myself, after a long search of many weeks on a problem that causes PHP-FPM to hang every 0-3 times a day. Both reload and restart fix this issue, but only reload is “soft” and non-user visible.

    Currently we have a monitoring script that checks a ping.php page every 10 seconds, and triggers the reload when the ping page fails.

    This works okay, but now a 2nd problem has arisen, sometimes causing problems every 1 in 20 requests. For some unknown reason, the config array (containing db connection info) is empty on those requests.

    For this second problem we’re looking at ulimit nofiles, as we think the xml config file of our app may be unreadable.

    Only the restart will fix this 2nd issue.

    Can you explain a bit more about the reasoning behind the reloads?

    Reply
    • All this stuff happened with me without any reason during PHP-FPM versions 5.2.xxx – 5.3.xxx since I moved 5.4.xxx branch all these what you’ve described above has gone. To be more precise now I’m using mostly Debian 7 and it’s PHP-FPM 5.4 version, which also allows to install php-apc straight via apt-get (no pear + cli) required. Also keep in mind that Debian rated php-fpm package as stable only since 5.4. I was waiting for stable version for 4 years since 2009. If you have ability to use this distro you may avoid all these issues.

      Reply
  6. I would like to know also the explanation of the php-fpm reload. Is it okay to reload this every 9 mins? I set my max server to 128 but I still received a “you may need to increase start_servers, or min/max_spare_servers” error.

    And also,why you need flush query cache?

    Reply
    • php-fpm reload also helps to clean the object cahce. In versions prior of php-fpm 5.4 restart interval not function properly, so I was doing it manually via cron.

      As for dynamic, I suggest you to use either static (if you got enough RAM) or ondemand value. Anyway number of children will be equal to the number of your CPU threads. So don’t spend you time on calculation of min/max_spare_servers.

      Reply
  7. Thank you a lot for sharing this with all people
    you really recognize what you’re talking about!

    Bookmarked. Kindly also discuss with my site =). We could have a
    link alternate contract between us

    encuentrocine.com site Newyorkwood.org portal

    Reply
  8. There aare considerations you need to think about if you want to sell your video content.
    Optimizing a web site primarily requires editing its content material and HTML and
    connected coding to both improve its relevance
    to particular key phrases and to remove barriers to the
    indexing activities of search engines like google. eval(ez_write_tag([[300,250],’brighthub_com-medrectangle-1′]));.

    Reply
  9. That is because the e-cigs are much more efficient inn keeping the
    blood nicotine levels above the threszhold that triggers a
    normal smoker to reach for another cigarette. Thiss smoke is generally a
    mixtire of a number of gases like carbon dioxide, carbon monoxide, carbon hydroxide,
    methane, propane, butane, isobutene, etc. After advertising on Craigslist, hhe soon had a huge
    customer base iin Cerritos and tthe surrounding areas.

    Reply

Leave a Comment.

This site uses Akismet to reduce spam. Learn how your comment data is processed.