Skip to main content

Friendica extremely slow since update to 2020.09 and 2020.09.1

At the beginning of this week I updated to 2020.09, and since then Friendica has run very slowly. When I reboot the VPS, it runs very fast for a short while, and then gets bogged down again. I use Plesk to monitor the VPS, which is only running Friendica, and almost the entire 4 GiB of memory is constantly in use. I can also see that the "Apache & PFP-FPM" memory usage is at 792 MiB, which I believe is high.

However, I cannot figure out why this is occurring and see no apparent errors in the logs. But also because I have had no errors in the 6 months I've been running Friendica, I don't know exactly what is "normal." Any assistance troubleshooting this would be greatly appreciated.
Please have a look at your FPM settings. Most likely the maximum number of processes is too low. Please check your FPM logfile.
Thank you very much for the replies. I saw a post from Steffen from 5 days ago, and now have changed my settings to match those (pasted again below). This is helping! Something must have changed in 2020.09, the default settings had always been fine up before. I will post again in the next day to confirm if the issue has been entirely resolved :)

PHP-FPM pool:
pm = dynamic
pm.max_children = 275
pm.start_servers = 24
pm.min_spare_servers = 6
pm.max_spare_servers = 42
pm.max_requests = 1250
@Matt Panhans I experience a similar increase in load. Especially I/O load. It started with/after the 2020.09 release when I remember correctly. But I'm not sure yet what exactly is causing this.

@Matt Panhans Please be careful with my settings. They're good for my machine with 64GB RAM. But they can be too high for a smaller sever. Also, with 275 max. FPM processes you need at least 350 possible DB connections or you will get problems/errors.
Thank you this is helpful. Does anyone have a suggestion on appropriate PHP-FPM setting for a small VPS (~4GB memory) running only Friendica?
@Matt Panhans Wait... I have a second server. It's a VPS. I'll have a look...
@Matt Panhans
These are my settings for the Friendica forum node (VPS):
pm.max_children = 60                                               pm.start_servers = 10                                             pm.min_spare_servers = 4                                           pm.max_spare_servers = 16
Wonderful. Last question- it looks like you do not specify the pm.max_requests here then? I will try these values out and see how it goes!
Those numbers are still really high for a small VPS, if you have your DB taking up half your RAM (which it probably should).

On my 8GB RAM VPS, I have pm.max_children at 25, which seems to have been satisfactory for awhile.
These settings seem to have resolved it on my little community node. Thank you for sharing them!
What is extremely slow at the moment is selecting a link to a post/comment from the notifications bell dropdown menu. It takes up to 30 seconds until the page is loaded.
It's funny where some are seeing issues. Direct links to posts, like from the notification bell, are working super-speedily for me.

What's slow is loading up /community, regardless whether it's local or global - it's very fast *after* the first page load, but can take 10 seconds (or more) to do the initial page load. But, that's about the slowest page load I'm seeing recently. (and I don't think it's new with the recent developments... it's always been one of the slower page loads as far as I can remember)
@Adam Disable the trending tags if they're active. The trending tags are a performance killer on the community pages.
Ah, excellent, thanks! I wasn't aware that one was taking so much time, but yes, turning it off helps a bunch.
I guess there was some background process running for several days. Since a couple of hours the heavy I/O load is gone und the page feels a lot snappier again. So, its was probably a local issue.
My issue appears to be resolve! I want to give a huge thank you to everyone for such prompt and helpful replies. I get better support here than I do from the IT department at my workplace!

My last thought on this issue - and I will leave to someone more technical to make this determination - is that it seems to me that there was an increase in Friendica's required resources/memory when moving from 2020.07 to 2020.09 that caused this issue. That's fine if expected, but if not it might be worth trying to understand the cause better, and perhaps get back to lower requirements. But I will leave to those more knowledgeable whether it's worth opening this as an issue on github.
Just letting you know from my end it's working really well / quickly now, I definitely noticed the previous lag. Thanks :)
I had a similar problem on my setup which I also started noticing after the last update of friendica. I'm also serving other service like Nextcloud on the same raspberry. I saw the following lines in my php log:

WARNING: [pool www] child 17597 exited on signal 11 (SIGSEGV) after 1326.443580 seconds from start

I found some hints to opcache being a potential cause for this kind of errors, which I have enabled. So I added, following a recommendation in the comments:

which seems to have solved the issue for me. Additionally I created a separate fpm pool only for friendica. Sometimes the issue took some time to appear, maybe I'm a bit to early with optimism.
Ok, I was too fast. My previous post solved the SIGSEGV issue but did not solve the slow reaction (up to the point where I get gateway timeouts) from friendica. This is my pool config:
pm = dynamic
pm.max_children = 15
pm.start_servers = 8
pm.min_spare_servers = 4
pm.max_space_servers = 15
pm.max_requests = 500

Playing with these values seams only to extend or shrink the time until friendica gets slow. Since I'm using a separate pool now it does not affect the other services any more.
Playing with these values seams only to extend or shrink the time until friendica gets slow.
I am having the exact same experience.
I opened an issue on Github yesterday (#9303) for this.
I wrote more details there as well.
Have you folks checked to see if you have an increase in traffic? How many entries/day do you have in your apache or nginx logs?

I have actually seen quite a drop in the number of hits to my instance this month, but perhaps some are getting more at present?
Good question. I do see a difference in traffic, in the Apache SSL/TLS access log. Sept 2 - Sept 4, 2020, it was logging an average of 32 entries per minute. On Sept 26, 2020, it was logging 171 entries per minute on average. For Sept 26, the entries are mostly many repeated "GET /.well-known/webfinger?" and "GET /friendica HTTP/1.0"

I guess the puzzle to me is what about updating to 2020.09 would have caused an increase in traffic? Any other info I can provide that would be helpful?
No. I saw these requests as well and I have to puzzle it out. Can have a look at the user-agent?
I'm uncertain if this answers your question, but the following three request lines were logged >46,000 times each on my server on Sept 26, 2020. The next most frequent request line was less than 5,000 times. (I am running Friendica on the domain

"GET /friendica HTTP/1.0 200 4303 - Friendica 'Red Hot Poker' 2020.09-1368;"
"GET /.well-known/webfinger? HTTP/1.0 200 836 - Friendica 'Red Hot Poker' 2020.09-1368;"
"GET /friendica HTTP/1.0 200 2511 - Friendica 'Red Hot Poker' 2020.09-1368;"
Yeah. Something seems to trigger massively repeated requests. I added some logging to my system. So when my system would at some time perform those requests as well, then I would see where it comes from.
Hmmm. That is my site. I’ll check my logs and config to see whats going on.
It would be interesting having a friendica log from that specific time frame.
Ok this is what i fount in my web server logs:
GET /friendica HTTP/1.1" 200 6252 "-" "Friendica 'Red Hot Poker' 2020.09-1368;" - is logged 107830 time from [25/Sep/2020:01:55:31 +0200] to the [26/Sep/2020:06:25:16 +0200]
Afther that is stops.

In my frendica log I have only 10 entrys for the 25. and 26. The log is only set up to log errors. Sorry. non that i can see relate to this.
@Matt Panhans
I noticed a marked increase in load times and slowness after update to .09 as well
Tweaked the FPM as suggested, made tiny difference.

Then I remembered the RELAYS

Am I correct that there is now a new/better relay function in FRND? As in: it's working properly where it wasn't before and possibly there are now many more requests and connections to federating instances?

I had added 3 ~ 4 extra relays to feed /community/global with more content, when I removed them again, the load times seemed to normalise.
The more relay servers you add, the more incoming connections you get. But I don't think that this is the reason for this specific issue. We already discovered some "ping pong" problem when two servers are requesting each others system account very often. This is already changed in the develop branch.
Just to report back. Since the update to 2021.01 I don't have to restart php constantly any more. #Friendica is faster and makes more fun than ever.

Thanks to all the developers and contributors.