Questions[opcache](server)

So for context, I am running PHP-FPM 7.4 on Ubuntu and I compiled it with ZTS. Then added the extensions Pthreads and ZSTD .

I am running a few hundred tasks and I need them to spawn new php processes as fast as possible, but after multiple tests I realized although my script was running in 0.001 secs, each time I spawned a new PHP Thread with Pthreads it added 0.01sec to the equation, which makes the time of each loop iteration multiply by 10 .

I am trying to find possible explanations for this for a couple of weeks already, and so far after doing multiple tests and tweaking a lot of settings the only thing that made sense is that this is a Pthreads Limitation by design .

Next logical step would be to update the code to the newer Parallel project but in the meantime I found something curious , when I enable opcache.enable_cli in PHP FPM config the issue is almost gone and most of the threads do not get delayed !

Sadly, this causes constant zend_mm_heap corruption / segmentation fault errors which from what I have read can't really be fixed (its either a zend or pthreads core issue) , but it would seem like it's a good start to understanding the root of the problem.

Anyone ?

Looking for some validation of my thought process here. I don't have the stats to hand to back up my claims, but have enough experience with web hosting with and without EFS to be comfortbale in making it.

A Joomla or WordPress PHP site being served entirely from EFS (e.g. the document root is in EFS) - making use of Apache and PHP-FPM will be slower than one that is served from EBS. This is because of the inherent latency of EFS for per-file operations. My understanding was that with Opcache enabled, all PHP scripts are compiled into Opcodes and stored in memory. if validate_timestamps is set to =0 then once a file is read from the file storage and compiled into Opcodes it is not then subsequently read.

What I don't understand is that even with the above set, requests to a page will still be slower on EFS. Even if the code base is held in memory.

Is my understanding incorrect and what else could be causing this latency?

We run a load balance with 3 Windows 2019 servers. One of them is the "master" one, serving SMB share to the other nodes, and it's also the server that Redis runs on (Redis used for sessions and cache). But this master srever is also a HTTP node.

Our problem is that on the master server, our website runs considerably faster than on the other nodes, it's 1 second of difference, pretty big.

In fact, the other (non-master) nodes have faster CPU, and on a PHP benchmark they perform better:

Master server (Redis, SMB, IIS):

PHP version : 7.4.16
Platform : WINNT
--------------------------------------
test_math                 : 0.396 sec.
test_stringmanipulation   : 2.818 sec.
test_loops                : 0.304 sec.
test_ifelse               : 0.374 sec.
--------------------------------------
Total time:               : 3.892 sec.

The other nodes:

PHP version : 7.4.16
Platform : WINNT
--------------------------------------
test_math                 : 0.311 sec.
test_stringmanipulation   : 2.334 sec.
test_loops                : 0.234 sec.
test_ifelse               : 0.191 sec.
--------------------------------------
Total time:               : 3.07 sec.

So it's definitely not a CPU issue, because the master node is slower. So it's either network, SMB or Redis issue.

I'm leaning towards Redis, because we use Wincache and Opcache for the nodes, so basically all .php files are cached locally for each node (SMB used only for revalidation every few minutes).

I do not know however how to find out where the bottleneck is exactly, and how to fix it.

Can anyone please help?

Thanks Alex

On our current monolithic LAMP servers we deploy via Capistrano and reload PHP-FPM at the end of a deployment, in order to flush the Opcode Cache.

Now, we're building a new webhosting cluster on AWS for dozens of PHP 7.0 and PHP 5.5 applications. The application-servers with PHP-FPM will autoscale horizontally. PHP code will be stored on an NFS server and static assets on Amazon S3.

How do I flush all the OpCaches after a deploy when I don't have a fixed list of hosts?

I've setup opcache with php-fpm but for some reason, it keeps manually restarting. I've not actioned a restart and I've set the force restart timeout to 2700 seconds (2 hours) so the cache shouldn't get cleared.

What causes a manual restart?

Array
(
    [opcache_enabled] => 1
    [cache_full] => 
    [restart_pending] => 
    [restart_in_progress] => 
    [memory_usage] => Array(
            [used_memory] => 58506120
            [free_memory] => 478364792
            [wasted_memory] => 0
            [current_wasted_percentage] => 0
        )
    [interned_strings_usage] => Array(
            [buffer_size] => 16777216
            [used_memory] => 5840560
            [free_memory] => 10936656
            [number_of_strings] => 3946503
        )
    [opcache_statistics] => Array(
            [num_cached_scripts] => 1553
            [num_cached_keys] => 3015
            [max_cached_keys] => 16229
            [hits] => 52263
            [start_time] => 1475655854
            [last_restart_time] => 1475662737
            [oom_restarts] => 0
            [hash_restarts] => 0
            **[manual_restarts] => 97**
            [misses] => 1553
            [blacklist_misses] => 0
            [blacklist_miss_ratio] => 0
            [opcache_hit_rate] => 97.114241117883
        )
)