Friendica extremely slow since update to 2020.09 and 2020.09.1
At the beginning of this week I updated to 2020.09, and since then Friendica has run very slowly. When I reboot the VPS, it runs very fast for a short while, and then gets bogged down again. I use Plesk to monitor the VPS, which is only running Friendica, and almost the entire 4 GiB of memory is constantly in use. I can also see that the "Apache & PFP-FPM" memory usage is at 792 MiB, which I believe is high.
However, I cannot figure out why this is occurring and see no apparent errors in the logs. But also because I have had no errors in the 6 months I've been running Friendica, I don't know exactly what is "normal." Any assistance troubleshooting this would be greatly appreciated.
Matt Panhans
Als Antwort auf Matt Panhans • • •Thank you very much for the replies. I saw a post from Steffen from 5 days ago, and now have changed my settings to match those (pasted again below). This is helping! Something must have changed in 2020.09, the default settings had always been fine up before. I will post again in the next day to confirm if the issue has been entirely resolved :)
PHP-FPM pool:
pm = dynamic
pm.max_children = 275
pm.start_servers = 24
pm.min_spare_servers = 6
pm.max_spare_servers = 42
pm.max_requests = 1250
Steffen K9 🐰
Als Antwort auf Matt Panhans • • •@Matt Panhans I experience a similar increase in load. Especially I/O load. It started with/after the 2020.09 release when I remember correctly. But I'm not sure yet what exactly is causing this.
@Matt Panhans Please be careful with my settings. They're good for my machine with 64GB RAM. But they can be too high for a smaller sever. Also, with 275 max. FPM processes you need at least 350 possible DB connections or you will get problems/errors.
Matt Panhans
Als Antwort auf Steffen K9 🐰 • • •Steffen K9 🐰
Als Antwort auf Matt Panhans • • •Steffen K9 🐰
Als Antwort auf Steffen K9 🐰 • • •These are my settings for the Friendica forum node (VPS):
Matt Panhans
Als Antwort auf Steffen K9 🐰 • • •Spencer
Als Antwort auf Steffen K9 🐰 • • •Steffen K9 🐰
Als Antwort auf Steffen K9 🐰 • • •Steffen K9 🐰
Als Antwort auf Steffen K9 🐰 • • •Matt Panhans
Als Antwort auf Matt Panhans • • •My issue appears to be resolve! I want to give a huge thank you to everyone for such prompt and helpful replies. I get better support here than I do from the IT department at my workplace!
My last thought on this issue - and I will leave to someone more technical to make this determination - is that it seems to me that there was an increase in Friendica's required resources/memory when moving from 2020.07 to 2020.09 that caused this issue. That's fine if expected, but if not it might be worth trying to understand the cause better, and perhaps get back to lower requirements. But I will leave to those more knowledgeable whether it's worth opening this as an issue on github.
Steffen K9 🐰
Unbekannter Ursprungsbeitrag • • •333
Als Antwort auf Matt Panhans • • •Steffen K9 🐰
Unbekannter Ursprungsbeitrag • • •github.com/friendica/friendica…
Martin Winkler
Als Antwort auf Matt Panhans • •I had a similar problem on my setup which I also started noticing after the last update of friendica. I'm also serving other service like Nextcloud on the same raspberry. I saw the following lines in my php log:
I found some hints to opcache being a potential cause for this kind of errors, which I have enabled. So I added, following a recommendation in the comments:
which seems to have solved the issue for me. Additionally I created a separate fpm pool only for friendica. Sometimes the issue took some time to appear, maybe I'm a bit to early with optimism.
Martin Winkler
Als Antwort auf Martin Winkler • •Playing with these values seams only to extend or shrink the time until friendica gets slow. Since I'm using a separate pool now it does not affect the other services any more.
Matt Panhans
Als Antwort auf Martin Winkler • • •I am having the exact same experience.
Matt Panhans
Als Antwort auf Matt Panhans • • •Martin Winkler
Als Antwort auf Matt Panhans • •Martin Winkler
Unbekannter Ursprungsbeitrag • •Matt Panhans
Unbekannter Ursprungsbeitrag • • •Good question. I do see a difference in traffic, in the Apache SSL/TLS access log. Sept 2 - Sept 4, 2020, it was logging an average of 32 entries per minute. On Sept 26, 2020, it was logging 171 entries per minute on average. For Sept 26, the entries are mostly many repeated "GET /.well-known/webfinger?" and "GET /friendica HTTP/1.0"
I guess the puzzle to me is what about updating to 2020.09 would have caused an increase in traffic? Any other info I can provide that would be helpful?
Matt Panhans
Unbekannter Ursprungsbeitrag • • •I'm uncertain if this answers your question, but the following three request lines were logged >46,000 times each on my server on Sept 26, 2020. The next most frequent request line was less than 5,000 times. (I am running Friendica on the domain sendero.social)
"GET /friendica HTTP/1.0 200 4303 - Friendica 'Red Hot Poker' 2020.09-1368; social.brevsprekk.net
"GET /.well-known/webfinger?resource=acct%3Afriendica%40sendero.social HTTP/1.0 200 836 - Friendica 'Red Hot Poker' 2020.09-1368; social.brevsprekk.net
"GET /friendica HTTP/1.0 200 2511 - Friendica 'Red Hot Poker' 2020.09-1368; social.brevsprekk.net
Geir Peter
Als Antwort auf Matt Panhans • • •Geir Peter
Als Antwort auf Matt Panhans • • •Ok this is what i fount in my web server logs:
GET /friendica HTTP/1.1" 200 6252 "-" "Friendica 'Red Hot Poker' 2020.09-1368; sendero.social - is logged 107830 time from [25/Sep/2020:01:55:31 +0200] to the [26/Sep/2020:06:25:16 +0200]
Afther that is stops.
In my frendica log I have only 10 entrys for the 25. and 26. The log is only set up to log errors. Sorry. non that i can see relate to this.
Falgn0n The Wizard
Als Antwort auf Matt Panhans • • •@Matt Panhans
I noticed a marked increase in load times and slowness after update to .09 as well
Tweaked the FPM as suggested, made tiny difference.
Then I remembered the RELAYS
Am I correct that there is now a new/better relay function in FRND? As in: it's working properly where it wasn't before and possibly there are now many more requests and connections to federating instances?
I had added 3 ~ 4 extra relays to feed /community/global with more content, when I removed them again, the load times seemed to normalise.
Martin Winkler
Als Antwort auf Matt Panhans • •Just to report back. Since the update to 2021.01 I don't have to restart php constantly any more. #Friendica is faster and makes more fun than ever.
Thanks to all the developers and contributors.