Okay, now this was a wild ride.
Background:
WIth the onslaught of AI scrapers, the server is getting bogged down. I still see it in our mission to serve them - and it's futile to try to block them anyway. At least they show where the software is slow ...
Really, there's three main areas that are popular with bots and hard to optimize for performance:
- PDF generation
- Server-side image manipulation (incl. the IIIF image view)
- Timelines
What I did the coding side:
- Reduce the number of simultaneous uploads to 2
- Implement a replacement of the default image view using OpenLayers. This way, we offer zoomable images with some added features (including a mini map of the image) that Mirador doesn't support by default. The hope is that this is better than Mirador for regular users. Importantly, it uses the simple image file (also served from a different server) rather than using the IIIF API. We thus save a lot of server-side image manipulation while ideally offering more.
On the sysad side:
- Separate the PHP-FPM settings for musdb and the frontend completely
- I also set up a separate worker to handle public PDF generation with minimal resources and priority
- The frontend generally has a reduced priority, while musdb has an elevated one. PDF generation has a very reduced one.
- The frontend settings are now much more restrictive than they were before, the musdb ones are yet more lenient in terms of memory usage and timeouts.
- The musdb process manager mode is set to
ondemand with a high number of maximum servers started. This may turn into a problem. Please let me know ASAP if the PHP-FPM server dies (http error 504). As long as not everybody logs in at the same time, we're going to be fine though. And we can handle the 20 uploads easily with that setting, I hope.
On the side I wrote a script to identify the number of requests to the PHP server. It's always interesting to see bot traffic in relation to regular traffic...
In Matomo, we see roughly 30k users a day for all of museum-digital. In museum-digital:global alone, we got around 1.1 million between 11 p.m. Thursday and 6 p.m. Friday. In the Germany-wide instance, there's around 1.4 million for the same time period.
Anyway, all the above didn't help. So (back to the sysad side) I ended up removing libvips (the image manipulation library we use), compiling it myself, failing again, installing it again.
And now it seems to work! Tried uploading the 10 files three times, and they all got uploaded without a problem after the re-installation.
We still got around 350 requests a second as we speak and I somehow don't trust the calm, especially looking at the server load. I'm curious of what it will look like tomorrow.