Optimizing large sites with WP-CLI

By default, ShortPixel Image Optimizer (SPIO) performs automatic optimization only when one of these pages is open in a new tab in the background. However, if you have a very large site that is constantly uploading images, you may want to use WP-CLI to customize the way optimization is run on your site. We have very useful documentation on using WP-CLI here: WP-CLI support for ShortPixel Image Optimizer – ShortPixel Knowledge Base

For most of our customers it is sufficient to run the wp spio bulk auto command once every few hours. However, for websites with a large number of images and daily uploads, problems can occur, such as PHP execution timeouts or re-executing a process before the same one is finished. For these websites, we have implemented a system based on "ticks" or "runs".

Requirement

First, add this code snippet to your functions.php file:

add_filter('shortpixel/medialibraryqueue/options', function ($options)
    {
        $options['numitems'] = 2; // number of items that will be pulled per tick during optimization
        $options['mode'] = 'wait';
        $options['process_timeout'] = 7000; // time between requests for the image (in milliseconds)
        $options['retry_limit'] = 30; // number of retries without errors before giving up
        $options['enqueue_limit'] = 200; // number of items added to the queue during preparation
        return $options;
    }
);

What is a "tick" or "run"?

Each "tick" or "run" is an action that sends a number of items (images plus the thumbnails that need to be optimized) to the API. The API either accepts the job as a new job or, if the request was made for the images already sent, checks to see if the process is complete and the results can be returned. If so, the API returns the results and the plugin replaces the original images with the optimized images. Most of the process time is (usually) spent downloading the results and processing the optimized images (saving backup copies, replacing the originals with the optimized images, etc.). The API stores the results for a maximum of 30 minutes. On average (depending on size and extras like WebP/AVIF), API-level processing takes between 10-20 seconds.

The queue also has a waiting time between "runs". It can't be predicted (unfortunately) how long a run will take. When only sending items to be optimized, it doesn't take long but when results are being downloaded this is dependent on the download speed, server disk speed, etc. If the waiting time for the process becomes higher than the interval of the cron job, there is a risk that multiple processes will be started in parallel on the server, which should be prevented as this can lead to slow throttling and possibly other problems.

Strategy

For a high traffic site where results are not needed immediately, we recommend an approach like this:

• Increase the number of items per run to 5 to 10 using the filter above. You will need to adjust the 'numitems'  parameter.
• Increase the process_timeout  parameter in the same filter to a value of more than 10 seconds, maybe even 30 seconds. This is the time the plugin waits before sending another request to the API to check if the images already sent for optimization are ready. You can check how long an average optimization takes and then adjust this parameter again.
• Add a cron job to run the queue in ticks (or runs) mode, not auto mode. You can add a minute-by-minute cron job with this command: wp spio run --ticks=5 --queue=media  . The default wait time is 3 seconds (see wp spio run  command parameters here) to leave room for other processes. This means that this command has a 5 ticks run with a minimum timespan of 15 seconds (if the items are in the queue, otherwise it terminates!). The other 45 seconds until the next cronjob runs are leeway for the server to process optimized images. This can all be tweaked depending on server load, download speed, etc.

Imagine you send 5 images per tick, 5 ticks, have a timeout of 30 seconds per image attempt, and the average optimization time is less than that number. This would mean that on a fast server, 25 images could be done in 30 seconds + time to download and process the files. The cron job could run every minute, which means that every minute a new process is started that does not count towards the execution limits.

The bottlenecks will always be the API requesting files from the server to process them, the API process, and downloading the images back to the server. For 25 images with an average of 10 thumbs, 250 images could be downloaded per minute (x2 if WebP is enabled) and it's up to your server to process those 250 files fast enough.

All of this can be tweaked. If you send and receive a lot of images, this will result in higher CPU peaks and network usage. If you send fewer files at once, you can spread the load more evenly, but it will take longer to optimize everything. If you have many users uploading images, spreading out the optimization process (basically a delay) could reduce resource usage.

If there is nothing to optimize, it will not run all ticks, but will exit directly. Therefore, in your case it is better to frequently request and tweak how much work the server can handle at the same time.

Frequently asked questions

Do I need to disable the automatic optimization in the plugin settings? 

There is no real need to disable the auto-optimization, but you can do it, because pretty much everything is processed by the cron jobs.

Do I need to set the compression method and other settings in the command parameters as well?

The settings are taken directly from the plugin settings, so you do not need to add anything to the WP-CLI command.

Are above numbers sufficient for all WordPress websites?

The above numbers in the Strategy section should be taken as a guideline for very large websites. For example, a website whose users upload about 3000 images per day. If you have a normal website that only uploads images once in a while, you do not even need to change the settings of WP-CLI. You can just stick to the wp spio bulk auto command (more information: WP-CLI support for ShortPixel Image Optimizer)

I have a lot of items in the Media Library that are already uploaded, and they are never optimized. What can I do?

1. Make sure that all settings you want are in place in the WordPress admin panel (thumbnails, next gen formats, exclusions, etc.)
2. Start a bulk optimization manually from your WordPress admin area.
3. Let the bulk optimization do all the calculations and let it run until it actually starts optimizing the images.
4. Then you can click on "Pause". This is necessary to ensure that all old elements that need to be processed are correctly added to the queue. Please note that this may take a while for large media libraries.
5. Set up a cron job to run every 5 minutes with the following command: wp spio run —ticks=20 —queue=media . This cron job will perform 20 operations in the 5 minutes between cron jobs.